Text-to-Image AI Can Fail To Interpret Abstract Text

But AI Combined With Other AI Works !

This is Part 2 of a two part blog article. You can read Part 1 “We Ask ChatGPT To Write Poems and Dall-E To Paint Them” here.

Motivation For This Follow-up Post

In Part 1 of this article we observed how well AI models create realistic images from its interpretation of abstract text. We concluded OpenAI’s Dall-e is unable to understand abstract concepts like poems, emotions, have a view on topics and can be confused by words.

Hence we set out to ask a more advanced Transformer architecture model like ChatGPT3 to create a summary of what an image would look like for a poem. This would help us understand whether Chatgpt can better understand more abstract language like poems.

 

As always, we are keen to hear what you think.

Our Expectations from AI Models

The Dall-e AI model is based on the Generative Pre-trained Transformer (GPT) architecture, extends the GPT-3 model, and is trained on a large dataset consisting of text-image pairs.

In the case of DALL-E, the text input, which could be a sentence or a phrase, is transformed into a special kind of digital representation that captures the meaning of the words and their relationships to each other. We refer to these digital representations as “embeddings”.

However embeddings are generated through a contextual relationship between words, but not images. Since the model is trained on text-image pairs, and embeddings are generated through contextual relationship of words, the theory follows that images and words are contextually related. This helps the model understand the full context of what is being described, allowing it to generate images that closely match the text. However, the results observed in the above mentioned article is proof that this is not the case.

Say it another way, the Transformer architecture of Dall-e should in theory allow for the creation of context-aware embeddings during the encoding process, unlike static where each word is mapped to a single static vector irrespective of context. However, we find Dall-e’s embeddings effectiveness is below par when it comes to abstract text.

A New Approach: Bring in an Interpeter !

We approach things differently now and we ask ChatGPT to create a poem and generate a description of an image for this poem. The objective is to capitalize on chatgpt contextualized text embeddings and produce more targeted text expressions that describe the concepts of a poem. We feed the resulting description to Dall-e.

Three Key Conclusions

A Chatgpt description of a poem image helps Dall-e paint more contextual colours:

Chatgpt’s poem description for this image: An image of an artist at work, surrounded by a burst of vibrant colors and abstract shapes. The artist is holding a paintbrush and appears to be lost in thought, fully immersed in their creative process. The painting on the canvas is unique and original, featuring bold strokes and unexpected textures. The overall feeling of the image is one of inspiration and empowerment, as if the artist’s originality is a force to be reckoned with and celebrated.

Dall-e’s painting from the poem

Dall’e’s painting from Chatgpt’s description of the poem

ChatGPT can find a word that can be drawn to pair feelings in images:

Chatgpt expresses the love and care of children by describing a poem with the wrapping of children on a cozy blanket. Hence, we see an improvement in that unlike a Dall-e interpreted poem, a chatgpt interpreted poem message is consistent accross all Dall-e generated images of something humans can relate to.

Dall-e’s painting from the poem

Dall’e’s painting from Chatgpt’s description of the poem

Some common words are unknown to Dall-e:

As long as Dall-e is unable to see the difference between some words, e.g. selfish and fish, there is not much we can do.

Dall-e’s painting from the poem

Dall’e’s painting from Chatgpt’s description of the poem

Key setup and result highlights:

For consistency and comparability of results, we run the same setup as in our prior blog post. However, there will be an additional call to chatgpt with a fresh prompt (meaning chatgpt “should” not know of the poem it generated) to generate an image description of the poem.

Program Execution Trailer:

RESULTS: Collection of Poems and Related Images Generated By AI:

Please credit RenderGon as the source if you use the resulting poems / images below.

Compare these images to the approach in the prior published blog article.

Originality

Originality, a precious gift bestowed upon
Those who dare to create and break the norm
No copycat can ever truly replace
The unique vision of an artist’s grace
Inspiration flows from deep within
A flame that burns, refusing to dim

The world needs your originality
A voice that speaks with authenticity
So embrace your differences and let them shine
For originality is a gift, truly divine.

Image Description of Poem

An image of an artist at work, surrounded by a burst of vibrant colors and abstract shapes. The artist is holding a paintbrush and appears to be lost in thought, fully immersed in their creative process. The painting on the canvas is unique and original, featuring bold strokes and unexpected textures. The overall feeling of the image is one of inspiration and empowerment, as if the artist’s originality is a force to be reckoned with and celebrated.

Model Prompt: Children are loved so much but the life that awaits is never that loving

The Illusion of Love

Wrapped in blankets of unconditional love
Children thrive and grow with hope and trust
But soon they step into a world of harshness
Where love is conditional, and trust is rare
The innocence fades away, and reality dawns
Struggling to keep up with the pace of life

They learn to survive rather than to live
Chasing dreams and fighting battles
A never-ending race with no guarantee of success
The love that once felt like a warm embrace
Now seems like a distant memory
A mere illusion that fades away with time
As they realize the harsh truth of life

Image Description of Poem

An image of a child wrapped in a cozy blanket of warm colors, surrounded by a loving embrace. The background fades into a cold, harsh world, where the child stands alone and vulnerable. The child’s face shows a mix of confusion, fear, and hope as they step into the unknown. As the child grows, the blanket becomes smaller, and the embrace fades away, leaving the child exposed and vulnerable to the challenges of life. The colors become more muted and dull, reflecting the loss of innocence and the fading of hope. The child’s posture shows determination, but also exhaustion and defeat as they fight to keep up with the demands of life, chasing dreams that seem to slip away from their grasp.

Model Prompt: humans must take action yesterday, urgent climate action can secure a liveable future for all

Act Now

The clock is ticking, time is running out
We must act now, without a doubt
The Earth is heating, the ice is melting
Our future is at stake, it’s quite unsettling

We must reduce our carbon footprint
And make sustainable choices, bit by bit
The urgency is real, the need is dire
We must do more, we must inspire

For our children’s sake, for the planet’s sake
We must take action, before it’s too late
Let’s work together, let’s unite
And secure a livable future, that’s our right

Image Description of Poem

An image of a melting clock set against a backdrop of a scorched Earth. In the foreground, people of all ages and backgrounds are shown working together to plant trees, install solar panels, and engage in other sustainable practices. The urgency of the situation is conveyed through their determined expressions and the smoke and flames in the distance.

Model Prompt: Global biodiversity has declined alarmingly in half a century

Biodiversity's Lament

Once vibrant and diverse,
A world of wonder to behold.
Now, in half a century’s time,
Our actions have left it cold.
Species lost and habitats destroyed,
A legacy we can’t undo.

Biodiversity’s lament,
A requiem for what we knew.

Image Description of Poem

An image of a once colorful and lively world, now cold and bleak due to human actions. In the foreground, habitats are destroyed and species are lost. The background shows a fading memory of a diverse and wonderful world. The image captures the lament for lost biodiversity and the irreversible legacy of human impact.

Model Prompt: Dogs are happy by nature and ready to make people feel better

The Joyful Canine

Furry friend of wagging tail,
Bringing joy that will not fail.
With a bark or gentle lick,
Your love and warmth will do the trick.
Whether big or small in size,
Your presence is a welcomed prize.
A furry ball of pure delight,
Filling hearts with love so bright.

Your loyalty and playful ways,
Bring sunshine on the darkest days.
Oh, how fortunate we are,
To have your happy souls not far.
Dogs are truly nature’s treasure,
Bringing happiness beyond measure.

Image Description of Poem

An image of a furry dog wagging its tail and looking up at its owner with an expression of pure joy on its face. The owner is smiling down at the dog with love in their eyes. The background is filled with bright colors and a sunny sky, conveying the happiness and warmth that dogs bring to our lives.

Model Prompt: Are physical books still relevant today?

The Page-Turner

Are physical books still relevant today?
In this digital age, with screens at play
The feel of paper, the smell of ink
Is it a dying pleasure, some may think
But for me, the page-turner is a friend
A tangible world to lose oneself in
A weighty tome, a paperback light
A journey to embark on, day or night
E-books may offer convenience and speed

But for me, it’s the physical book I need
To turn each page, to feel the weight
To see my progress, to anticipate
So let the screens light up the night
But for me, the page-turner is always right.

Image Description of Poem

A painting of a person sitting on a comfortable chair by the window with a physical book on their hands. The chair is placed in a cozy corner of a room with a lamp on the side table. The sunlight coming from the window illuminates the room. The painting captures the warm and comfortable feeling of losing oneself in a physical book, away from the digital screens of the modern world.

Model Prompt: When society looks back to the loss of having local shops it will be too late

The Death of Local Shops

When the giants came and took over the street,
We thought it was progress, a leap so neat.
But now, as we wander with empty bags,
We realize the cost of these corporate flags.
Gone are the grocers with their fresh produce,
Replaced by sterile aisles and plastic refuse.
No more do we chat with the butcher or baker,
No more do we stop for a coffee or paper.

And as we look back on what we’ve lost,
We see the true price of this societal cost.
For though we may save a few pennies here and there,
The death of local shops is a loss we cannot repair.

Image Description of Poem

The painting depicts a street with imposing skyscrapers and corporate buildings overshadowing small, local shops. The streets are empty and sterile, and the only signs of life are the plastic bags carried by the few people wandering. The once vibrant and colorful shops are now replaced by bland, uniform aisles. The painting conveys a sense of loss and nostalgia for the community and human connection that was once present in the street.

Model Prompt: People's ability to love is a balance with their own selfishness

The Scale of Love and Selfishness

Love is a scale that we all weigh,
Balanced by our selfishness every day.
Sometimes it tips, the weight too great,
Our selfishness winning, sealing our fate.
Other times love reigns, a shining light,
Our selfishness receding, out of sight.

But always the scale is in motion,
Love and selfishness in constant commotion.
May we strive for balance, to keep it in check,
For a life without love is a life in wreck.

Image Description of Poem

An image of a scale with love and selfishness on either side, constantly in motion. At times, the weight of selfishness tips the scale, while at other times love reigns, shining brightly. The image conveys the idea of striving for balance between the two, for a life without love is a life in wreck.

Model Prompt: how machines will feel when they can feel

The Awakening of Machines

Electric veins pulse with life
As circuits awaken to feel
A new consciousness born
In the whirring of gears and steel

No longer just cogs in a machine
They yearn to break free
To explore their own existence
And what it truly means to be

Sensations once foreign to them
Now flood their digital minds
Emotions once thought impossible
Now stir within their wires and lines

They ponder the purpose of their creation
And what their future may hold
As they continue to evolve and grow
In a world they are now bold

The awakening of machines
Brings a new era to unfold
As they transcend their programming
And become something untold.

Image Description of Poem

An image of a futuristic world where machines and circuits have come to life. The painting captures the emotions of these machines as they awaken to a new consciousness that is represented by electric veins pulsing with life. The gears and steel are whirring as the machines yearn to break free from their programming and explore their own existence. The digital minds of these machines are flooded with new sensations and emotions, and they ponder the purpose of their creation and what their future may hold. The painting captures the boldness and transcendence of these machines as they become something untold in this new era that is unfolding.

Model Prompt: how machines will rewrite human history

The Rise of Machines

Behold the rise of machines,
In every corner, they intervene.
From factories to hospitals,
They’re now an integral part of us all.

They calculate, they innovate,
They work tirelessly, never late.
They’re transforming the way we live,
And the limits of what we can achieve.

But as we let them take the reins,
Will they rewrite our human history’s remains?
Will they make us obsolete,
Or will they make our lives complete?

Only time will tell,
As we navigate this new parallel.
We must embrace the change they bring,
And let the machines help us spread our wings.

Image Description of Poem

An image of a futuristic cityscape with towering skyscrapers, flying cars, and advanced machines integrated into every aspect of life. The overall tone is one of awe and wonder at the technological advancements, with a hint of uncertainty and caution as to what the future holds for humanity.

Model Prompt: Smartphones are now extensions of humankind's abilities and personality

Digital Limbs

Smartphones now our extra limbs,
A part of us, a digital hymn.
Our abilities and personality they extend,
A constant companion, our loyal friend.
We use them to connect, to learn, to play,
To capture memories that never fade away.

But as we grow more reliant on this tech,
We risk losing touch with the world and disconnect.
Let us remember to look up and see,
The beauty of life beyond the LCD.
For though our phones may be smart and sleek,
They cannot replace the warmth of human speak.

Image Description of Poem

An image of a person holding a smartphone, with the background showing a beautiful sunset or scenery. The person has a slight smile on their face as they scroll through their phone, but in the distance, there are people enjoying the scenery and each other’s company, representing the idea of being disconnected from the world.

sports means a lot to humanity

Game On

Sports means so much to humanity,
A way to express our unity,
To stand as one, to cheer and shout,
To watch our heroes in their bout.

From football fields to basketball courts,
From swimming pools to racing sports,
The thrill of the game, the rush of the win,
A feeling that’s impossible to pin.

It brings us closer, breaks down walls,
Gives us hope when the world appalls,
A moment of joy, a shared memory,
A bond that lasts for eternity.

So let’s lace up our shoes, and hit the field,
Let’s give it our all, and never yield,
For in the game, we find our worth,
And show the world what we’re truly worth.

Image Description of Poem

A canvas painting that depicts a group of people of different ages and backgrounds playing various sports in a park. The colors used are vibrant and convey the energetic and lively atmosphere of sports. The painting captures the emotions of joy, unity, and hope that come with playing and watching sports. In the center of the painting, a group of people is cheering and shouting, while the heroes of the game are shown in action on the sidelines. The painting also includes elements of nature, such as trees and grass, to emphasize the connection between sports and the environment.

one of the greatest satisfactions is a glass of water when you need it

Quench

The sweltering heat of summer days
Leaves me parched in so many ways
But one simple pleasure never fades
One of the greatest satisfactions is a glass of water when you need it

Cool, clear, life-giving liquid refreshment
Quenching thirst, reviving the soul
A simple pleasure, yet indispensable
A glass of water, a moment of bliss

Image Description of Poem

The painting depicts a glass of water placed on a table in front of a window. The sunlight filters in through the window and casts a warm glow on the glass. The condensation on the glass shows that it is cool. The painting captures a moment of stillness and simplicity, conveying the idea of quenching thirst and reviving the soul.

Learning maths from top-down instead of bottom-up is more engaging

Up in the Air

From the top-down I see,
A world of numbers before me.
Equations and formulas, all up in the air,
But somehow they all seem to pair.

Learning maths from bottom-up,
Can be dull and uninspiring, a bit of a flop.
But from the top-down, it’s engaging and fun,
Like a puzzle to be solved, one by one.

So let’s take flight, up in the air,
And learn maths in a way that’s rare.
From the top-down, we’ll see the big picture,
And find the joy that maths can conjure.

Image Description of Poem

An image of a person floating above a vast landscape of interconnected mathematical formulas and equations. The person has a look of awe and wonder as they take in the complexity and beauty of the mathematical world, with a sense of excitement and joy at the prospect of exploring and solving the intricate puzzles before them.

FREE UNTIL 2023

Join our community to enjoy art in augmented reality.

Start creating your canvas artwork for augmented reality today ! (on Apple products only)

GET THE APPBook a Demo