How to Finetune ChatGPT on Your Use Case?

With the craze around ChatGPT the concept of LLMs or Large Language Models has been added in everyone’s new vocabulary. There isn’t a meeting or Zoom call where you don’t hear the words but what about ChatGPT and how can we use it. To use ChatGPT or similar technologies one of the most important aspects is to understand how to control and train these models on what works for your organization and a bit part of that is fine tuning LLMs. However, to fine-tune these models to suit specific use cases, we must understand how to approach this task effectively. In this article, we will discuss the importance of fine-tuning and how it is done in the context of OpenAI models in particular and LLMs in general.

Understanding Large Language Models

What are Large Scale Language Models?

Large scale language models or LLM’s are NLP models that are trained on large datasets to improve their language processing capabilities. These models can process and analyze text data to perform a variety of tasks such as language translation, sentiment analysis, and content generation. The most notable examples of large language models include ChatGPT (OpenAI), Turing NLG (Microsoft), Gopher, Chichilla (Deepmind), T5, MT5 (Google), Ernie 3.0 (Baidu) which are all trained on billions of tokens.

What are the Benefits of Large Language Models?

The benefits of large language models are numerous, including the ability to generate high-quality text, improved language processing, and enhanced accuracy in predicting language-based outcomes. These models can also learn from and adapt to new data, improving their performance over time and powering technologies such as Chatbots, Conversational AI, Email bots, personalized Voice bots, SEO and even Natural language coding.

Fine-Tuning Pretrained Language Models

What is Fine-Tuning Pretrained Language Models?

Fine-tuning pretrained language models involves taking an existing language model that has already been trained on a large dataset and retraining it on a smaller, domain-specific dataset. This process fine-tunes the model to understand and produce language relevant to a specific domain or use case.

Fine-tuning can:

1. Significantly improve the accuracy of responses by having a better understanding of the task at hand and hence the accuracy of the responses.
2. Change and match the tone of the responses to your organization's requirements eg. professional and short or casual and friendly.

As per OpenAI’s own findings, Fine-tuning GPT-3 improves accuracy by 2 to 4x over what’s possible with prompt design.

Can you fine tune ChatGPT?

As of July 2023 you can do the following to fine tune Generative AI platforms including and like ChatGPT:

1. Prompt based fine tuning: Prompt engineering is a growing field and prompts are a great and easy way of fine tuning the model based on your use case. For fine tuning based on prompts you first need to give some examples of your different outputs with both the question and the expected response. This will enable the system to understand what you expect and automatically fine tune based on this. You can use the system prompts for this type of fine tuning so that you do not have to include the same message each time.
2. Instruction based fine tuning: If prompt based fine tuning is not sufficient (which in many cases it isn’t especially when you are working with proprietary data like medical analysis) then you need to create your own customized GPT model. Here we recommend to use the Llama2 model with QLoRa based fine tuning. (You can read this article for step by step details) https://medium.com/@ud.chandra/instruction-fine-tuning-llama-2-with-pefts-qlora-method-d6a801ebb19 . This is the method that is ideal for companies that want to teach and fine tune weights of the model for particular tasks rather than making it a general machine.
3. Token based fine tuning: In certain cases though this also will not be enough where you need more specificity and understanding of your domains language and intricacies. A good example of this was BloombergGPT where they fine tuned the entire token system.

Fine-Tuning vs. Feature Extraction

What is the Difference between Fine-Tuning and Feature Extraction?

Fine-tuning and feature extraction are two approaches to reusing existing models for a specific task. Fine-tuning involves training the entire model on new data, while feature extraction only trains a new classifier on top of the existing model's pre-extracted features. Fine-tuning is typically more effective when the task is more complex or requires the model to learn more about the domain, while feature extraction can be more efficient for simpler tasks.

Why Do We Need Fine-Tuning?

Fine-tuning is essential because it allows us to reuse existing, powerful models for specific use cases. Instead of training an entirely new model from scratch, we can save time and resources by fine-tuning a pretrained model for our specific needs. This approach can also improve the accuracy and performance of the model, especially in situations where labeled training data is limited.

How to Fine-Tune a Large Language Model

1. Preparing for Fine-Tuning:
   Before fine-tuning LLMs, like ChatGPT, it is important to prepare the necessary data. This involves understanding the task at hand, choosing the right dataset, and formatting the dataset.
1. Understanding the task at hand: Understanding the specific task you want LLMs like ChatGPT to perform is crucial. For example, if you want to fine-tune for sentiment analysis, you need to understand what sentiment analysis involves, how many different sentiments do you want to classify the text into and how LLMs like ChatGPT can be used for this task.
2. Choosing the right dataset: Once you understand the task, you need to choose a dataset that is relevant to the task. The dataset should contain examples of the type of text that LLMs like ChatGPT will be generating. For example, if you are fine-tuning for sentiment analysis, you need a dataset that contains examples of text with sentiment labels with all the sentiments you want the language model to understand.
3. Formatting the dataset: The dataset needs to be properly formatted so that it can be used for fine-tuning. This involves separating the text into input and output sequences and ensuring that the labels are properly formatted.
2. Fine-Tuning Techniques:
   There are several fine-tuning techniques that can be used to optimize LLMs like ChatGPT for specific tasks. These include few-shot learning, zero-shot learning, continual learning, and multi-task learning.
1. Few-Shot Learning: Few-shot learning involves fine-tuning LLMs like ChatGPT on a small amount of data. This technique is useful when you have limited labeled data for the task at hand.
2. Zero-Shot Learning: Zero-shot learning involves fine-tuning LLMs on a task that it has not been specifically trained for. This technique is useful when you want to use LLMs for a task that is similar to the ones it has been trained on like content writing.
3. Continual Learning: Continual learning involves fine-tuning LLMs like ChatGPT over time as new data becomes available. This technique is useful when you want to keep LLMs like ChatGPT up-to-date with new information and continue to improve its model based on new data. This is extremely useful to increase accuracy but if not implemented correctly it can also drop accuracy.
4. Multi-Task Learning: Multi-task learning involves fine-tuning LLMs like ChatGPT on multiple related tasks simultaneously. This technique is useful when you want to optimize LLMs like ChatGPT for multiple tasks at once.

How to get started with fine-tuning?

1. Setting up the environment: Before fine-tuning LLMs like ChatGPT, you need to set up the environment by installing the necessary libraries and dependencies.
2. Running the fine-tuning code: Once the environment is set up, you can begin running the fine-tuning code. This involves feeding the formatted dataset into the code and specifying the fine-tuning technique to use. The code will then fine-tune LLMs like ChatGPT on the dataset and generate a fine-tuned model.
   This is how the dataset jsonl file would like.

{"prompt": "", "completion": "<ideal generated text>"}

{"prompt": "", "completion": "<ideal generated text>"}

{"prompt": "", "completion": "<ideal generated text>"}
....

You would have to take care to keep each record less than 2048 tokens. Once, you have the dataset ready, run it through the OpenAI command-line tool to validate it. Openai tools fine_tunes.prepare_data - <LOCAL_FILE>

You can also pass files in CSV, TSV, XLSX, JSON or JSONL format to this tool and it will help you convert it into a fine-tuning ready dataset.

3. Train a new fine-tuned model:
   Run the below command from the command line program to train your fine-tuned model.
   openai api fine_tunes.create -t <TRAIN_FILE_ID_OR_PATH> -m <BASE_MODEL>
   Replace the filename and choose a model name to base your model on.
   
   
4. Evaluating the fine-tuned model: After the fine-tuning process is complete, you need to evaluate the fine-tuned model to ensure that it is performing well on the task at hand. This involves using a separate dataset to test the performance of the fine-tuned model. If the model is not performing well, you may need to adjust the fine-tuning technique or hyperparameters and repeat the process.

Conclusion: The fine-tuning process can be iterative, and it may take several rounds of fine-tuning and evaluation to optimize LLMs like ChatGPT for a specific task. However, with the right approach and techniques, fine-tuning LLMs like ChatGPT can lead to impressive results and help unlock its full potential.

If you would like to see how this can be applied to your organization, reach out to us at technology@enterprisebot.ai or to our sales team at sales@enterprisebot.ai.