test out private GPT on AWS EC2

https://medium.com/@dminhk/3-easy-steps-to-run-localgpt-on-amazon-ec2-de79d336d4ee

this article turns out to be the 1st link when I googled for ‘setup gpt on ec2’, so I followed the steps there to do my 1st experiment.

git clone https://github.com/SamurAIGPT/privateGPT.git

This is rather a very intuitive private GPT project inspired by the more well known https://github.com/imartinez/privateGPT, and it is really very easy to try out on commodity hardware.

the repo has two folders, client and server; the client is node.js web app listening at 3000, and the server is a Python Flask project listening at 5000. You just need to install the relevant dependencies and run both client and server in order to launch it.

notes on ec2:

• you might need to add your IP address to security group for port 3000 and 5000
• you might need to modify the various localhost:3000 and localhost:5000 endpoint in in the javascript code to point to your EC2’s public IP address.
• use the latest version of node and npm

The web UI has a button to download a default model (https://gpt4all.io/models/ggml-gpt4all-j-v1.3-groovy.bin) and allows you to upload a local document; the model is only a few gigabytes, so it only takes a few minutes to download.

The speed to query is rather quite slow. On a EC2 with 8 cpu cores and 16 GB of RAM, it takes probably 1 mins to answer some simple question.