GPUs on Fly.io are available to everyone!

GPUs are now available to everyone! We know you’ve been excited about wanting to use GPUs on Fly.io and we’re happy to announce that they’re available for everyone. If you want, you can spin up GPU instances with any of the following cards: Ampere

Fly.io logo
Fly.io
•
Xe Iaso
•2 min read•intermediate•
--
•View Original
Large Language ModelsOllama

Overview

Fly.io has announced the availability of GPU instances for everyone, enabling users to leverage powerful GPUs for applications like large language models, text transcription, and image generation. The article details the types of GPU cards available and provides a quick guide on how to set up a GPU instance.

What You'll Learn

1

How to spin up GPU instances on Fly.io using Ampere A100 and Lovelace L40s cards

2

Why using GPUs can enhance applications involving large language models and image generation

3

How to configure an Ollama server on Fly.io in seconds

Key Questions Answered

What types of GPU cards are available on Fly.io?
Fly.io offers several GPU cards including the Ampere A100 with 40GB and 80GB, and the Lovelace L40s with 48GB. These powerful GPUs can be utilized for various applications such as large language models and image generation.
How can I set up a GPU instance on Fly.io?
To set up a GPU instance on Fly.io, you need to change the 'vm.size' for your app to one of the available GPU types. An example configuration is provided in the article, which includes setting the app name, region, and GPU type.
What is the process to deploy an Ollama server on Fly.io?
Deploying an Ollama server on Fly.io involves configuring your app with the appropriate settings, including specifying the GPU type and internal port. The article provides a code example to illustrate this setup process.
When can we expect lower-cost A10 GPUs on Fly.io?
Fly.io is working on making lower-cost A10 GPUs available in the coming weeks, with updates to be provided when they are ready for use.

Technologies & Tools

Some links below are affiliate links. We may earn a commission if you make a purchase.

Cloud Platform
Fly.io
Used for deploying GPU instances and applications.
AI/ML Framework
Ollama
Utilized for running large language models.

Key Actionable Insights

1
Utilize Fly.io's GPU instances to enhance your applications that require heavy computational power, such as AI/ML tasks.
By leveraging the available GPUs, developers can significantly improve the performance of applications that involve large datasets or complex computations.
2
Experiment with the provided code example to quickly set up an Ollama server on Fly.io.
This setup allows for rapid deployment of large language models, making it easier to integrate advanced AI functionalities into your applications.
3
Stay tuned for the upcoming A10 GPU offerings to access more cost-effective options for GPU computing.
These lower-cost GPUs will provide additional flexibility for developers looking to optimize their cloud computing expenses.

Common Pitfalls

1
Failing to configure the correct 'vm.size' can lead to unsuccessful deployment of GPU instances.
Ensure that the specified GPU type matches one of the available options to avoid deployment errors.

Related Concepts

Large Language Models
Image Generation
Text Transcription
Cloud Computing