AI Image Generator and Story Generation App using FastAPI, Groq and Replicate

Project Introduction: AI Image Generator and Story Creator

The AI Image Generator and Story Creator is a web application that leverages advanced AI technologies to provide users with an interactive platform for generating images and stories based on audio prompts. The application utilizes FastAPI for the backend, enabling efficient handling of requests and responses, while the frontend is built with HTML, CSS (DaisyUI and Tailwind CSS), and JavaScript for a responsive user experience. This application leverages llama-3.1–70b for prompt generation ,black-forest-labs/flux-1.1-pro for image generation and llava-v1.5–7b vbision model for story cretion via Groq and Replicat.AI repectively.

Key Features:

1. Audio Recording and Transcription: Users can record their voice prompts, which are then transcribed into text using speech recognition technology.

2. Image Generation: Based on the transcribed text, the application generates detailed image prompts and creates corresponding images using the Replicate API.

3. Image Downloading: Users can download the generated images to their local devices.

4. Story Generation: The application can generate engaging stories based on the images created, providing a narrative context to the visual content.

5. User-Friendly Interface: The application features a clean and intuitive interface, making it easy for users to interact with the various functionalities.

Technologies Used:

• Backend: FastAPI, Groq, Replicate.ai, SpeechRecognition
• Frontend: HTML, CSS (DaisyUI, Tailwind CSS), JavaScript
• Image Processing: Pillow for image handling
• Asynchronous Operations: aiohttp and aiofiles for efficient file handling and network requests

This project serves as a demonstration of integrating multiple AI services into a cohesive application, allowing users to explore the creative possibilities of AI-generated content

Detailed Explanation of the Codebase:

1. Frontend (HTML/JavaScript):

• The application uses a single HTML page (index.html) with a responsive design using DaisyUI and Tailwind CSS.
• The page contains sections for audio recording, transcription, prompt generation, image generation, and story generation.
• The JavaScript file (script.js) handles user interactions and communicates with the backend API.

2. Backend (FastAPI) :

• The main application is defined in app/main.py.
• It uses FastAPI to create a web server with various endpoints:

— a. /: Serves the main HTML page.

— b. /transcribe: Transcribes audio to text.

— c. /generate_prompt: Generates an image prompt from text using Groq’s LLM.

— d. /generate_image: Generates an image using Replicate’s Flux model.

— e. /download_image: Downloads and saves the generated image.

— f. /generate_story_from_image: Generates a story based on the image using Groq’s LLaVA model.

— g. /download/{filename}: Serves the downloaded image file.

3. Key Features:

• Audio recording and transcription
• Text-to-image prompt generation
• Image generation from prompts
• Story generation from images
• Image downloading and saving

4. External APIs:

• Groq: Used for text generation (tweaked prompts and stories)
• Replicate AI: black-forest-labs/flux-1.1-pro model used for image generation
• Necessary Packages to be Installed:

fastapi
uvicorn
jinja2
python-multipart
pydantic
python-dotenv
groq
replicate
SpeechRecognition
pydub
aiohttp
aiofiles
Pillow

You can install these packages using pip:

pip install fastapi uvicorn jinja2 python-multipart pydantic python-dotenv groq replicate SpeechRecognition pydub aiohttp aiofiles Pillow

Execution Instructions:

• Set up environment variables: Create a .env file in the root directory with the following content:

GROQ_API_KEY=your_groq_api_key_here
REPLICATE_API_TOKEN=your_replicate_api_token_here

Replace the placeholder values with your actual API keys.

• Ensure you have all the necessary files in place:
• — app/main.py
• — app/config.py
• — app/utils.py
• — templates/index.html
• — static/css/styles.css
• — static/js/script.js
• Run the FastAPI server: Navigate to the directory containing app/main.py and run:

uvicorn app.main:app - reload

• Access the application:
• — Open a web browser and go to http://127.0.0.1:8000
• Using the application:
• — a. Click “Start Recording” and speak your prompt.
• — b. Click “Stop Recording” when finished.
• — c. The audio will be transcribed automatically.
• — d. Click “Generate Image Prompt” to create a detailed prompt.
• — e. Click “Generate Image” to create an image based on the prompt.
• — f. Use the “Download Image” button to save the generated image.
• — g. Click “Generate Story” to create a story based on the generated image.

Note: Ensure you have proper internet connectivity, as the application relies on external APIs for various functionalities.

This application demonstrates a complex integration of various AI technologies, including speech recognition, language models, and image generation, all wrapped in a user-friendly web interface.

The FastAPI UI as illustrated below

AI Image Generator Application

Speak your prompt

• Start Recording
• Stop Recording
• Transcription(Transcribed Text): a beautiful Indian model walking down the Runway Ram as a part of fashion show
• Create a new Prompt based on the transcribed text to generate images
• — Generated Prompt: “Generate a highly realistic image of a stunning Indian model walking down the iconic Runway Ram as part of a high-end fashion show. The model, a 22-year-old Indian woman with long dark hair, dark brown eyes, and flawless skin, should be dressed in an exquisite, intricately embroidered lehenga choli with gold and silver sequins, traditional Indian attire, and pair it with heeled stilettos. Her outfit is designed with intricate embroidery and fine stitching. Emphasize the elegant pleats, sparkling fabrics, and her elegant poise and confident stride. Incorporate elaborate jewelry pieces like beads, gold bangles, and necklaces on her hands, neck and one side styled hairstyle. Lighiting effects play an enormous role, set up warm stage headlights that emphasize the model’s attire and light the whole surrounding with mild bluish tone. Camera angles should display the outfit’s details entirely. Desired scene perspective is front full body shot of model in the middle, catwalk around her lit with intense golden hue lights illuminating from within.”

Generated Image

Story Generated from the Image

The stunning sight of this pageant queen in a gold and silver sequined outfit with lehenga skirt and bling-drop earrings is enough to leave any spectator mesmerized. Sashaya Gnanavel is prominently featured in the foreground, walking confidently down the runway to captivate the audience at the show. Her stylish attire, complemented by an elegant pearl necklace, draws the attention of everyone present. The collection showcases vibrant colors and sparkling embroideries, which add to the overall visual appeal of the event. Sashaya’s confidence and beauty in the spotlight are a true testament to her talent and dedication to the fashion industry. The dazzling effect created by her makeup, jewels, and exquisite outfit helps set the stage for an extraordinary showcase of design and craftsmanship. This compelling scene encapsulates magic and opulence, where spectators are left in awe by the sheer exquisiteness of it all.

Code Implementation

Create Virtual Environment

To create a virtual environment using Python’s venv module, follow these steps:

• Open your terminal or command prompt.
• Navigate to your project directory (where you want to create the virtual environment). You can use the cd command to change directories. For example:

cd path/to/your/project

• Create a virtual environment by running the following command:

python -m venv venv

• This command creates a new directory named venv in your project folder, which will contain the virtual environment (on windows)
• Activate the virtual environment:

venv\Scripts\activate

• folder structure

• utils.py

import base64
import os
from pydub import AudioSegment

def save_audio(audio_data):
    # Decode the base64 audio data
    audio_bytes = base64.b64decode(audio_data.split(",")[1])
  
    # Save the audio to a temporary file
    temp_file = "temp_audio.webm"
    with open(temp_file, "wb") as f:
        f.write(audio_bytes)
  
    # Convert WebM to WAV
    audio = AudioSegment.from_file(temp_file, format="webm")
    wav_file = "temp_audio.wav"
    audio.export(wav_file, format="wav")
  
    # Remove the temporary WebM file
    os.remove(temp_file)
  
    return wav_file

def text_to_speech(text):
    # Implement text-to-speech functionality if needed
    pass

• main.py

"""
    1. Record audio through their microphone
    2. Transcribe the audio to text
    3. Generate an image prompt using the Groq Llama3 model
    4. Generate an image using the Replicate.ai Flux model
    5. Display the generated image
    6. Download the generated image
    The application uses DaisyUI and Tailwind CSS for styling, providing a dark mode interface. The layout is responsive and should work well on both desktop and mobile devices.
Note: You may need to adjust some parts of the code depending on the specific APIs and models you're using, as well as any security considerations for your deployment environment.

"""
from fastapi import FastAPI, Request, HTTPException
from fastapi.templating import Jinja2Templates
from fastapi.staticfiles import StaticFiles
from fastapi.responses import JSONResponse, FileResponse
from pydantic import BaseModel
import speech_recognition as sr
from groq import Groq
import replicate
import os
import aiohttp
import aiofiles
import time
from dotenv import load_dotenv
load_dotenv()
from .utils import text_to_speech, save_audio
from PIL import Image
import io
import base64
import base64


## Function to encode the image
def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')

app = FastAPI()

app.mount("/static", StaticFiles(directory="static"), name="static")
templates = Jinja2Templates(directory="templates")

## Initialize Groq client with the API key
GROQ_API_KEY = os.getenv("GROQ_API_KEY")
if not GROQ_API_KEY:
    raise ValueError("GROQ_API_KEY is not set in the environment variables")
groq_client = Groq(api_key=GROQ_API_KEY)

class AudioData(BaseModel):
    audio_data: str

class ImagePrompt(BaseModel):
    prompt: str

class PromptRequest(BaseModel):
    text: str

## Add this new model
class FreeImagePrompt(BaseModel):
    prompt: str
    image_path: str

@app.get("/")
async def read_root(request: Request):
    return templates.TemplateResponse("index.html", {"request": request})

@app.post("/transcribe")
async def transcribe_audio(audio_data: AudioData):
    try:
        # Save the audio data to a file
        audio_file = save_audio(audio_data.audio_data)

        # Transcribe the audio
        recognizer = sr.Recognizer()
        with sr.AudioFile(audio_file) as source:
            audio = recognizer.record(source)
        text = recognizer.recognize_google(audio)

        return JSONResponse(content={"text": text})
    except Exception as e:
        raise HTTPException(status_code=400, detail=str(e))

@app.post("/generate_prompt")
async def generate_prompt(prompt_request: PromptRequest):
    try:
        text = prompt_request.text
        # Use Groq to generate a new prompt
        response = groq_client.chat.completions.create(
            messages=[
                {"role": "system", "content": "You are a creative assistant that generates prompts for realistic image generation."},
                {"role": "user", "content": f"Generate a detailed prompt for a realistic image based on this description: {text}.The prompt should be clear and detailed in no more than 200 words."}
            ],
            model="llama-3.1-70b-versatile",
            max_tokens=256
        )
        generated_prompt = response.choices[0].message.content
        print(f"tweaked prompt:{generated_prompt}")
        return JSONResponse(content={"prompt": generated_prompt})
    except Exception as e:
        print(f"Error generating prompt: {str(e)}")
        raise HTTPException(status_code=400, detail=str(e))

@app.post("/generate_image")
async def generate_image(image_prompt: ImagePrompt):
    try:
        prompt = image_prompt.prompt
        print(f"Received prompt: {prompt}")

        # Use Replicate to generate an image
        output = replicate.run(
            "black-forest-labs/flux-1.1-pro",
            input={
                "prompt": prompt,
                "aspect_ratio": "1:1",
                "output_format": "jpg",
                "output_quality": 80,
                "safety_tolerance": 2,
                "prompt_upsampling": True
            }
        )
      
        print(f"Raw output: {output}")
        print(f"Output type: {type(output)}")
      
        # Convert the FileOutput object to a string
        image_url = str(output)
      
        print(f"Generated image URL: {image_url}")
      
        return JSONResponse(content={"image_url": image_url})
    except Exception as e:
        print(f"Error generating image: {str(e)}")
        raise HTTPException(status_code=400, detail=str(e))

@app.get("/download_image")
async def download_image(image_url: str):
    try:
        # Create Output folder if it doesn't exist
        output_folder = "Output"
        os.makedirs(output_folder, exist_ok=True)

        # Generate a unique filename
        filename = f"generated_image_{int(time.time())}.jpg"
        filepath = os.path.join(output_folder, filename)

        # Download the image
        async with aiohttp.ClientSession() as session:
            async with session.get(image_url) as resp:
                if resp.status == 200:
                    async with aiofiles.open(filepath, mode='wb') as f:
                        await f.write(await resp.read())

        # Return the filepath and filename
        return JSONResponse(content={
            "filepath": filepath,
            "filename": filename
        })
    except Exception as e:
        print(f"Error downloading image: {str(e)}")
        raise HTTPException(status_code=400, detail=str(e))

class StoryRequest(BaseModel):
    filepath: str
    filename: str

@app.post("/generate_story_from_image")
async def generate_story_from_image(content: StoryRequest):
    try:
        image_path = content.filepath
        print(f"Image path: {image_path}")
        # Check if the file exists
        if not os.path.exists(image_path):
            raise HTTPException(status_code=400, detail="Image file not found")

        # Getting the base64 string
        base64_image = encode_image(image_path)

        client = Groq()

        chat_completion = client.chat.completions.create(
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "Generate a clear,concise,meaningful and engaging cover story for a highly acclaimed leisure magazine based on the image provided. The story should keep the audience glued and engaged and the story should bewithin 200 words."},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": f"data:image/jpeg;base64,{base64_image}",
                            },
                        },
                    ],
                }
            ],
            model="llava-v1.5-7b-4096-preview",
        )

        story = chat_completion.choices[0].message.content
        print(f"Generated story: {story}")
        return JSONResponse(content={"story": story})
    except Exception as e:
        print(f"Error generating story from the image: {str(e)}")
        raise HTTPException(status_code=400, detail=str(e))

@app.get("/download/{filename}")
async def serve_file(filename: str):
    file_path = os.path.join("Output", filename)
    return FileResponse(file_path, filename=filename)

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)

• script.js

let mediaRecorder;
let audioChunks = [];

const startRecordingButton = document.getElementById('startRecording');
const stopRecordingButton = document.getElementById('stopRecording');
const recordingStatus = document.getElementById('recordingStatus');
const transcription = document.getElementById('transcription');
const generatePromptButton = document.getElementById('generatePrompt');
const generatedPrompt = document.getElementById('generatedPrompt');
const generateImageButton = document.getElementById('generateImage');
const generatedImage = document.getElementById('generatedImage');
const downloadLink = document.getElementById('downloadLink');
const generateStoryButton = document.getElementById('generateStory');
const generatedStory = document.getElementById('generatedStory');

startRecordingButton.addEventListener('click', startRecording);
stopRecordingButton.addEventListener('click', stopRecording);
generatePromptButton.addEventListener('click', generatePrompt);
generateImageButton.addEventListener('click', generateImage);
generateStoryButton.addEventListener('click', generateStory);

async function startRecording() {
    const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
    mediaRecorder = new MediaRecorder(stream);

    mediaRecorder.ondataavailable = (event) => {
        audioChunks.push(event.data);
    };

    mediaRecorder.onstop = sendAudioToServer;

    mediaRecorder.start();
    startRecordingButton.disabled = true;
    stopRecordingButton.disabled = false;
    recordingStatus.textContent = 'Recording...';
}

function stopRecording() {
    mediaRecorder.stop();
    startRecordingButton.disabled = false;
    stopRecordingButton.disabled = true;
    recordingStatus.textContent = 'Recording stopped.';
}

async function sendAudioToServer() {
    const audioBlob = new Blob(audioChunks, { type: 'audio/webm' });
    const reader = new FileReader();
    reader.readAsDataURL(audioBlob);
    reader.onloadend = async () => {
        const base64Audio = reader.result;
        const response = await fetch('/transcribe', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({ audio_data: base64Audio }),
        });
        const data = await response.json();
        transcription.textContent = `Transcription: ${data.text}`;
        generatePromptButton.disabled = false;
    };
    audioChunks = [];
}

async function generatePrompt() {
    const text = transcription.textContent.replace('Transcription: ', '');
    const response = await fetch('/generate_prompt', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({ text: text }),
    });
    const data = await response.json();
    generatedPrompt.textContent = `Generated Prompt: ${data.prompt}`;
    generateImageButton.disabled = false;
}

async function generateImage() {
    const prompt = generatedPrompt.textContent.replace('Generated Prompt: ', '');
    const response = await fetch('/generate_image', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
        },
        body: JSON.stringify({ prompt: prompt }),
    });
    const data = await response.json();
    generatedImage.src = data.image_url;
  
    // Download the image and get the filepath
    const downloadResponse = await fetch(`/download_image?image_url=${encodeURIComponent(data.image_url)}`);
    const downloadData = await downloadResponse.json();
  
    // Store the filepath and filename for later use
    generatedImage.dataset.filepath = downloadData.filepath;
    generatedImage.dataset.filename = downloadData.filename;

    // Set up the download link
    downloadLink.href = `/download/${downloadData.filename}`;
    downloadLink.download = downloadData.filename;
    downloadLink.style.display = 'inline-block';
}

async function generateStory() {
    const imagePath = generatedImage.dataset.filepath;
    const filename = generatedImage.dataset.filename;
  
    if (!imagePath || !filename) {
        generatedStory.textContent = "Error: Please generate an image first.";
        return;
    }

    try {
        const response = await fetch('/generate_story_from_image', {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
            },
            body: JSON.stringify({ filepath: imagePath, filename: filename }),
        });
        if (!response.ok) {
            throw new Error(`HTTP error! status: ${response.status}`);
        }
        const data = await response.json();
      
        // Display the generated story
        generatedStory.textContent = data.story;
      
        // Make sure the story container is visible
        document.getElementById('storyContainer').style.display = 'block';
    } catch (error) {
        console.error('Error:', error);
        generatedStory.textContent = `Error: ${error.message}`;
    }
}

// Modify the download link click event
downloadLink.addEventListener('click', async (event) => {
    event.preventDefault();
    const response = await fetch(downloadLink.href);
    const blob = await response.blob();
    const url = window.URL.createObjectURL(blob);
    const a = document.createElement('a');
    a.style.display = 'none';
    a.href = url;
    a.download = response.headers.get('Content-Disposition').split('filename=')[1];
    document.body.appendChild(a);
    a.click();
    window.URL.revokeObjectURL(url);
});

• style.css

body {
    background-color: #1a1a2e;
    color: #ffffff;
}

.container {
    max-width: 1200px;
}

#imageContainer {
    min-height: 300px;
    display: flex;
    align-items: center;
    justify-content: center;
    background-color: #16213e;
    border-radius: 8px;
}

#generatedImage {
    max-width: 100%;
    max-height: 400px;
    object-fit: contain;
}

• index.html

<!DOCTYPE html>
<html lang="en" data-theme="dark">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>AI Image Generator</title>
    <link href="https://cdn.jsdelivr.net/npm/daisyui@3.7.3/dist/full.css" rel="stylesheet" type="text/css" />
    <script src="https://cdn.tailwindcss.com"></script>
    <link rel="stylesheet" href="{{ url_for('static', path='/css/styles.css') }}">
</head>
<body>
    <div class="container mx-auto px-4 py-8">
        <h1 class="text-4xl font-bold mb-8 text-center">AI Image Generator</h1>
        <div class="grid grid-cols-1 md:grid-cols-2 gap-8">
            <div class="card bg-base-200 shadow-xl">
                <div class="card-body">
                    <h2 class="card-title mb-4">Speak your prompt</h2>
                    <button id="startRecording" class="btn btn-primary mb-4">Start Recording</button>
                    <button id="stopRecording" class="btn btn-secondary mb-4" disabled>Stop Recording</button>
                    <div id="recordingStatus" class="text-lg mb-4"></div>
                    <div id="transcription" class="text-lg mb-4"></div>
                    <button id="generatePrompt" class="btn btn-accent mb-4" disabled>Generate Image Prompt</button>
                    <div id="generatedPrompt" class="text-lg mb-4"></div>
                    <button id="generateImage" class="btn btn-success" disabled>Generate Image</button>
                </div>
            </div>
            <div class="card bg-base-200 shadow-xl">
                <div class="card-body">
                    <h2 class="card-title mb-4">Generated Image</h2>
                    <div id="imageContainer" class="mb-4">
                        <img id="generatedImage" src="" alt="Generated Image" class="w-full h-auto">
                    </div>
                    <a id="downloadLink" href="#" download="generated_image.png" class="btn btn-info" style="display: none;">Download Image</a>
                </div>
            </div>
        </div>
        <!-- Add this new section after the existing cards -->
        <div class="card bg-base-200 shadow-xl mt-8">
            <div class="card-body">
                <h2 class="card-title mb-4">Generate Story from Image</h2>
                <button id="generateStory" class="btn btn-primary mb-4">Generate Story</button>
                <div id="storyContainer" class="mb-4">
                    <p id="generatedStory" class="text-lg"></p>
                </div>
            </div>
        </div>
    </div>
    <script src="{{ url_for('static', path='/js/script.js') }}"></script>
</body>
</html>

Conclusion

The AI Image Generator and Story Creator project successfully integrates various AI technologies to create an interactive web application that allows users to generate images and stories based on audio prompts. By leveraging FastAPI for the backend and modern frontend technologies, the application provides a seamless user experience.

Key Takeaways:

1. Integration of AI Models: The project demonstrates how to integrate multiple AI models, including Groq for text generation and Replicate for image generation, to create a cohesive application that enhances user creativity.
2. User Interaction: The application allows users to interact through voice commands, making it accessible and user-friendly. The ability to record audio, transcribe it, and generate content based on that input showcases the potential of voice-driven applications.
3. Dynamic Content Generation: By generating images and stories dynamically based on user input, the application highlights the capabilities of AI in content creation, providing users with unique and personalized outputs.
4. Responsive Design: The use of DaisyUI and Tailwind CSS ensures that the application is visually appealing and responsive, catering to users on various devices.
5. Future Enhancements: The project can be further enhanced by incorporating additional features such as user authentication, saving user-generated content, and expanding the range of AI models used for different creative tasks.

Overall, this project serves as a comprehensive example of how to build an AI-powered web application that combines audio processing, image generation, and storytelling, paving the way for innovative applications in the creative domain.

References

• FastAPI Documentation: FastAPI is a modern web framework for building APIs with Python. It is designed to be easy to use and fast.
• Pydantic: Pydantic is used for data validation and settings management using Python type annotations.
• Groq:Groq is a platform for building and deploying AI models. It provides APIs for text generation and other AI tasks.
• Replicate: Replicate is a platform that allows you to run machine learning models in the cloud. It provides APIs for various models, including image generation.
• SpeechRecognition: SpeechRecognition is a library for performing speech recognition, with support for several engines and APIs.
• Pillow: Pillow is a Python Imaging Library (PIL) fork that adds image processing capabilities to your Python inter
• JavaScript Fetch API: The Fetch API provides a modern way to make network requests in JavaScript.
• HTML5 Audio API: The HTML5 Audio API allows you to play audio files in web applications.
• DaisyUI: DaisyUI is a component library for Tailwind CSS that provides pre-designed components.
• Tailwind CSS: Tailwind CSS is a utility-first CSS framework for creating custom designs without having to leave your HTML.