Geek Out Time: Simple Local Testing of Llama 3 on Its Release, Gemma, and Mistral Mistral

Upon the release of Llama 3, I conducted tests on three models locally on my 8G RAM M1 Macbook: gemma:2b (I would have preferred to use gemma:7b, but encountered a ‘model not found’ error in ollama. Therefore, not a fair play. Pls leave a note below if you know why), llama3:8b, and Mistral:7.3b. This blog presents the test questions along with their answers and evaluates the correctness and detailed performance of each model. I utilized the Python-based ollama library to query the models and assess them based on execution time, peak memory usage, and response accuracy. Below is the Python script used for the tests.

import time import psutil from memory_profiler import memory_usage import ollama model_names = [ 'gemma:2b' , 'llama3' , 'mistral' ] questions = [ "What is the capital of France?" , "Explain the theory of relativity." , "Who wrote 'Pride and Prejudice'?" , "How does photosynthesis work?" ] def ask_questions ( model, questions ): for question in questions: print ( f"Question: {question} " ) def model_call (): return ollama.chat(model=model, messages=[{ 'role' : 'user' , 'content' : question}]) start_time = time.time() memory_and_response = memory_usage(proc=model_call, interval= 0.01 , retval= True , max_usage= True ) peak_memory, response = memory_and_response end_time = time.time() answer = response[ 'message' ][ 'content' ] if 'message' in response and 'content' in response[ 'message' ] else "No valid response" elapsed_time = end_time - start_time print ( f"Answer: {answer} \nTime taken: {elapsed_time: .2 f} seconds\nPeak memory used: {peak_memory: .2 f} MB\n" ) print ( "--------------------------------------------------" ) if __name__ == '__main__' : for model_name in model_names: load_start = time.time() model = model_name load_end = time.time() load_time = load_end - load_start print ( f"\nTesting model: {model_name} \nModel loading time: {load_time: .2 f} seconds" ) ask_questions(model, questions) Test Questions and Model Responses I cannot use question libraries like Glue, SQuAD, etc., because they take too long to run. Therefore, I presented each model with the same small set of questions to evaluate their performance and accuracy :

This is an excerpt — the full article continues on Medium.