About Integrate Ollama API

[sfiannaca]

Just for fun, I played with HTTP calls and crafted this raw and rough Dart file to access the Ollama API for retrieving models and play with the Stream Chat API.

//Dart File
import 'package:http/http.dart' as http;
import 'dart:async';
import 'dart:convert';

class ApiConfig {
String type;
String model;
bool enabled;
String urlGen;
String urlModels;

ApiConfig({
required this.type,
required this.model,
required this.enabled,
required this.urlGen,
required this.urlModels,
});
}

ApiConfig apiConfig = ApiConfig(
type: "OLLAMA",
model: "llama2",
enabled: false,
urlGen: "http://192.168.1.102:8000/api/generate", // CHANGE IP AND PORT
urlModels: "http://192.168.1.102:8000/api/tags", // CHANGE IP AND PORT
);
void main() async {
gen(String prompt) async* {
final request = http.Request('POST', Uri.parse(apiConfig.urlGen));
request.headers['Content-Type'] = 'application/json';
request.body = jsonEncode({"model": apiConfig.model, "prompt": prompt});
final response = await http.Client().send(request);

print('Response status: ${response.statusCode}');
if (response.statusCode == 200) {
  final controller = StreamController<Map<String, dynamic>>();
  response.stream.transform(utf8.decoder).listen((data) {
    if (data.isNotEmpty) controller.add(jsonDecode(data));
  });
  yield* controller.stream;
  
} else {
  throw Exception('Error on response');
}

}
getModels()async{
final response = await http.get(Uri.parse(apiConfig.urlModels));
return jsonDecode(response.body);
}

final models = await getModels();
print(" Models " + models.toString());

final streamController = StreamController<Map<String, dynamic>>();
final apiResponseStream = gen("Why is the sky blue?");

apiResponseStream.listen(
(data) {
print('Received data: $data');
},
onError: (error) {
print("Error: $error");
},
onDone: () {
print("API request completed.");
streamController.close();
},
);

await streamController.stream.drain();
print("All data processed.");
}

Also, there is another third-party library that can be used: github.com/davidmigloz/langchain_dart

[danemadsen]

Looks good. I'm still trying to think how i should implement the ollama API, id like it if the user was able to control model settings like seed, temp, etc from the app without messing with it on the computer / server

[alxspiker]

Other ideas is to send the parameters to Olama if it is on first connect to the server, or actively being used and parameters were changed, this could allow the same UI to be used.

[danemadsen]

That sounds promising. I'm not really familiar with their API yet. Plus i want to improve the stability of my app first before i implement another major change like that

[alxspiker]

Oh I see, I never looked into Ollama until now.

[danemadsen]

Yeah, it would be good to support it to draw more attention to this project.

[sfiannaca]

Yeah, the params can be set via API too.
ref

final options =
{
"temperature": 0.7,
"num_gpu": 20,
"num_thread": 4,
"repeat_penalty": 1.1,
"num_ctx": 2048,
"mirostat": 0,
"mirostat_eta": 0.1,
"mirostat_tau": 5.0,
"num_gqa": 8,
"num_predict": 42,
"top_k": 40,
"top_p": 0.9,
"tfs_z": 1.0,
//"stop": "Assistant:",
"seed": 42
};
request.body = jsonEncode(
{
"model": apiConfig.model,
"prompt": prompt,
"options": options
}
);

[sfiannaca]

That sounds promising. I'm not really familiar with their API yet. Plus i want to improve the stability of my app first before i implement another major change like that

It makes sense to prioritize improving the app's stability

[sfiannaca]

I'm writing this just to inform you in case you're not already aware.
You can add the following line below the [Service] section within the 'ollama.service' file located at /etc/systemd/system for linux

Environment="OLLAMA_HOST=0.0.0.0:8000"

This modification allows Ollama to listen on all network interfaces (0.0.0.0), enabling sharing within your local environment, so you can test it on mobile too

[danemadsen]

Yeah thats good. Currently i have it working in a very basic manner, no model params are passed yet and the example prompts cant be used yet.

[sfiannaca]

I came across the suggestion to make api models configurable in the comments and shared an example in an attempt to assist. You can find the example here: main...sfiannaca:Maid:dev-api.

However, I hadn't noticed that you had already added the switch inside the drawer, which is also a good way.

[danemadsen]

yeah its probably going to be one of the most impactful setting in the app so ill probably keep it in the drawer for the foreseeable future. The other option would be to put in on the app bar and mimic how the ChatGPT app used to be with switch between 3.5 to 4

[danemadsen]

I'm waiting for ollama/ollama#991 to be pushed through to do a proper release with Ollama support.