Sentiment Analysis of Caption, News and Fiction text generation experiments

From: text generation editor research (UAL London 2020 see credits).
What happens when writers use a computer text generator to write articles, giving them only an image prompt.

Go to Index of AI research

Sentiment Analysis

The Study had three text generation and editing tasks, to make a Caption,  a News article, and a Fiction story, using the same image prompt of a dog and man.

Summary

This shows differences in Positive sentiment affect between the three different experiments. Most positive at the start, by the end, a balanced neutral view had become evident.

Fear was shown in the first Caption experiment, but was lost in the second News and third Fiction, showing acclimatisation by experience. Generally Negative scores were low.

By Fiction experiment, the third and final, only Sadness was left.

Comment

This shows a learning process during the three experiments.

All results are cautioned by the strong ‘Tentative’ score 0.91 and lack of any ‘Confidence’ scores over 0.5.

Details – Method

For overall sentiment of responses, all text feedback was summed into a total text field per respondent.

Text comment fields were:

Caption, News and Fiction Experiments (3 fields);

Questions 1 (2 fields), 2-6a (5 fields), 6b Fake news (1 field).

This gives 11 feedback text samples per respondent (not all were filled). These are summed vertically in Excel to give the overall text block per column/field.

Texts were also summed per respondent, horizontally in Excel.

NLP and IBM Tone Analyser

NLP (natural language processing) allows computer analysis of text blocks. For this volume of text, the online IBM Tone Analyser (See References at bottom) was used. Writing a custom analyser was outside the scope of the study. Human grading was not possible due to the size of texts when totalled, however human analysis is used for the summary of texts.

IBM Tone Analysis gives a rating for:

Anger Fear Sadness Joy Analytical Confident Tentative

(Coded on data as Ang, A, F, S, J, A, C, T.)

The “most prevalent tones that are detected for each utterance” are shown at a document level, and sentence level.

The document level analysis has scores, and the sentence level (which shows lower occurrences) was added in brackets. This gives a results for example, where the detected tones have a numeric score over 0.5. Scores: <.5 None; =>.5 – .75 Mid; >.75 Strong

At document level, each tone if found, has a score .5-.1.0

Lower graded tones (placed in brackets in data) are placed at the 0.25 level

Examples

J,S,A,T(C,F) – Joy, Sadness, Analytic and Tentative have scores > .5 (and C Confidence, F Fear have lower occurrences only, between >0 and < 0.5)

Ang,F,S() – Ang is Anger, Fear and Sadness have scores > .5 (no others over 0)

F(S) –Fear scores > .5 (Sadness between >0 and < 0.5)

In my data display, a ‘lower occurrence’ (bracketed) is scored at 0.25

Confidence and Tentative are general attitudes shown in the text.

Results

All respondents summed

Caption experiment – all feedback comments

Graph - IMB TA Caprion.png

With motivated stakeholders as respondents, there are high scores for ‘Analytical’ and ‘Tentative’. ‘Confidence’ did not appear at all as a document level tone in all 82 people, and occasionally as a sentence level tone.

Using the highest score amongst ‘Anger’, ‘Fear’ and ‘Sadness’ as Negative, and using ‘Joy’ as Positive, shows a higher degree of positive response.

Fear is evident, but at a low level. Sadness is the strongest of the negative reactions.

Positive is about 20% more than Negative. (Significance.)

News experiment – all feedback comments

Graph - IMB TA News.png

More overall positive result than Caption experiment.

Fear has gone, low levels for Anger and Sadness Negative tones. Positive is about 42% more than Negative.

Fiction experiment – all feedback comments

Graph - IMB TA Fiction.png

Sadness at its highest level. Not Positive or Negative. Anger and Fear do not appear.

Summary

This shows differences in Positive sentiment affect between the three different experiments. Most positive at the start, by the end, a balanced neutral view had become evident.

Fear was shown in the first Caption experiment, but was lost in the second News and third Fiction, showing acclimatisation by experience. Generally Negative scores were low.

By Fiction experiment, the third and final, only Sadness was left.

Comment

This shows a learning process during the three experiments.

All results are cautioned by the strong ‘Tentative’ score 0.91 and lack of any ‘Confidence’ scores over 0.5.

References

IBM Tone Analyser

Please see full Report  for further statistics (tba).

Guess the original classics – Born Into Slavery

Born into Slavery

This is a game to see if you can identify the original story author in the generated text below. It has a new title: Born Into Slavery. The generated text is followed by the classic story which was used as the prompt for the computer generation. For technical details see end.


 

Born Into Slavery

What would occur when the daughter came for a visit? Her daughter could not wish to come here but if she had to she was certain to be disappointed.

When her eldest daughter was in her early thirties, Mr. Smith thought it might be advisable to take her away with him to her own country. The time had been long since passed; what had gone before was now forgotten. She was quite happy at home, though she knew the consequences of her leaving the family.

Mrs. Smith, who lived on her own farm, was the more interested of the two, as she had a son she loved. She would rather she had one of her own daughters than, with him, a son. As it was she was obliged to take him along, though with a slight advantage in health and strength. The child, however, was well behaved, and was very fond of her.

The old couple would have had a child when Mrs. Smith was dead; the latter would not have died till Mr. Smith, as they thought, had been ninety years old, and with her, after him. The child, as they said, would live till old Mr. Smith had died; if that should prove impossible, they would have had an heir to one of the first families to the west of the river.

Mrs. Smith was still an old woman, she remembered; her eyes had become dim, her hands and feet were weak, her feet were dull and bent, her hands wrinkled and loose. A change from her old condition, she thought, would have been too much for her now. She had always been happy in her old condition, though always in the present mode.

This story made her do something she thought necessary for the little girl, and she went into the hall and sat down in the great old chair opposite to Mrs. Smith

A woman’s strength, she thought, was her knowledge of her father’s affairs. She knew that as long as her children could walk with her, they could not be married.

Her husband knew that too; he knew where he was going, what he was going to do, and was never the same afterwards. There was a very large number of children, her daughters, living abroad, in the most irregular manners. He had grown somewhat deaf, with a peculiar difficulty about their voices.

And Mrs. Smith never liked it when she was in any degree unhappy; she could not bear to see her men, much less her children, so miserable. Every time they came she wanted to have a chat with them. She was a woman of very little self-control, and she would have felt an extraordinary strain in talking to men in whom she was so anxious to be connected. She had grown quite aware of the dangers of these visits. She felt that she had the moral to avoid them. It was the same with her husband.

One of the dangers she had to observe was whether they should try to give her good advice on an important subject; then she could say nothing but good things about them. And she did not give you any reason why.

And so he took her into another house, where she found him dead. Here is the explanation of one of our own writers, that was born into slavery, and who has since been made a public writer.

In the first place, we say that his wife knew, but she did not have sufficient excuse for going back to him. For she had been married to his eldest son, whom he kept with her, but she had never had a son, not that she was less prone to be the subject of any kind of cruelty than any other woman.

She was a woman well grounded, and could be trusted not to become a mistress or to marry a knave. Now, the reason why Master Smith was the property of Mr. Smith was, that there were several houses there, called for the use of this one Mr. Smith, and he was not there, but was there at a different time; so he did not know him or know of his own family.

But of course this also became a reason for the destruction of his father’s property; for those other houses were to be used by Mr. Smith at a later time, and hence there was no excuse to lose them. And he knew of the condition of his father’s property, and that he had been made a public writer.

Now he said to Master Smith,
“What is the reason of your going to me at this time, that I do not go to him myself?”

“Well, I have said to him before, I do not go to him, because my father was in debt to you.”

“How did I know it?”

“He had a contract of sale of my house, for you had a lease for the premises. You and your father made a bargain, and did not pay; but he does me to make good my debt. He gave me this lease, but left the property to me.”

“What can I do for it?”

“I will not go with you.”

And Mr. Smith went away in his ship; and while he was doing this, he heard the voice of his father’s son, who was living in the ship, saying to him,

“Your father has made you a public writer, at a time when I should have told you everything that you ought to know about me; and he is coming to me to sell my property for your property, and that we must not meet there till the day after the feast.”

As it happened, at the end of his talk with Mr. Smith, who would then have come to him, he heard of the destruction of Mr. Smith’s house, and of the damage done, for a great many months afterwards, to the houses belonging to his son, because of that man’s writing; and Mr. Smith went away in his ship, that there might be no necessity of his telling Mr. Brown that he might not receive anything from him from him without reason.

And it was, at that time, a great pleasure to him to have the ship and his house in the same box, and he said nothing of it.


Original


Pride and Prejudice

Jane Austen 1813

It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife.

However little known the feelings or views of such a man may be on his first entering a neighbourhood, this truth is so well fixed in the minds of the surrounding families, that he is considered the rightful property of some one or other of their daughters.

“My dear Mr. Smith,” said his lady to him one day, “have you heard that Manorfield Park is let at last?”

Mr. Smith replied that he had not.

“But it is,” returned she; “for Mrs. Long has just been here, and she told me all about it.”

Mr. Smith made no answer.

“Do you not want to know who has taken it?” cried his wife impatiently.

You want to tell me, and I have no objection to hearing it.”

This was invitation enough.

“Why, my dear, you must know, Mrs. Long says that Manorfield is taken by a young man of large fortune from the north of England; that he came down on Monday in a chaise and four to see the place, and was so much delighted with it, that he agreed with Mr. Morris immediately; that he is to take possession before Michaelmas, and some of his servants are to be in the house by the end of next week.”

“What is his name?”

“Bingley.”

“Is he married or single?”

“Oh! Single, my dear, to be sure! A single man of large fortune; four or five thousand a year. What a fine thing for our girls!”

“How so? How can it affect them?”

“My dear Mr. Smith,” replied his wife, “how can you be so tiresome! You must know that I am thinking of his marrying one of them.”

“Is that his design in settling here?”

“Design! Nonsense, how can you talk so! But it is very likely that he may fall in love with one of them, and therefore you must visit him as soon as he comes.”

“I see no occasion for that. You and the girls may go, or you may send them by themselves, which perhaps will be still better, for as you are as handsome as any of them, Mr. Bingley may like you the best of the party.”

“My dear, you flatter me. I certainly have had my share of beauty, but I do not pretend to be anything extraordinary now. When a woman has five grown-up daughters, she ought to give over thinking of her own beauty.”

“In such cases, a woman has not often much beauty to think of.”

“But, my dear, you must indeed go and see Mr. Bingley when he comes into the neighbourhood.”

“It is more than I engage for, I assure you.”

“But consider your daughters. Only think what an establishment it would be for one of them. Sir William and Lady Lucas are determined to go, merely on that account, for in general, you know, they visit no newcomers. Indeed you must go, for it will be impossible for us to visit him if you do not.”

“You are over-scrupulous, surely. I dare say Mr. Bingley will be very glad to see you; and I will send a few lines by you to assure him of my hearty consent to his marrying whichever he chooses of the girls; though I must throw in a good word for my little Lizzy.”

“I desire you will do no such thing. Lizzy is not a bit better than the others; and I am sure she is not half so handsome as Jane, nor half so good-humoured as Lydia. But you are always giving her the preference.”

“They have none of them much to recommend them,” replied he; “they are all silly and ignorant like other girls; but Lizzy has something more of quickness than her sisters.”

“Mr. Smith, how can you abuse your own children in such a way? You take delight in vexing me. You have no compassion for my poor nerves.”

“You mistake me, my dear. I have a high respect for your nerves. They are my old friends. I have heard you mention them with consideration these last twenty years at least.”

“Ah, you do not know what I suffer.”

“But I hope you will get over it, and live to see many young men of four thousand a year come into the neighbourhood.”

“It will be no use to us, if twenty such should come, since you will not visit them.”

“Depend upon it, my dear, that when there are twenty, I will visit them all.”

Mr. Smith was so odd a mixture of quick parts, sarcastic humour, reserve, and caprice, that the experience of three-and-twenty years had been insufficient to make his wife understand his character. Her mind was less difficult to develop. She was a woman of mean understanding, little information, and uncertain temper. When she was discontented, she fancied herself nervous. The business of her life was to get her daughters married; its solace was visiting and news.


Notes

Prompt was 10-20 words from the original. GPT-2 system by Fabrice Bellard – see the site below for credits etc.
Made on my creativity app Story Live – please visit.

Text from Gutenburg free classic ebooks

A certain amount of cherry-picking of the most grammatical and interesting generated parts, but no actual editing (moving around of words or new words). Identifying names are changed.

More of these will be posted, see the index.

Fake News – what do real writers think about it? AI research

I examined in August 2020 research (UAL London see credits) what happens when writers use a computer text generator to write articles, giving them only an image prompt. There is a summary of the Study in the Index.

Go to Index of AI research

Fake News – (or ‘fake news’) what do real writers think about it?

Summary 

  • People who have a lower general opinion of text generation, and lower wordcounts for feedback, don’t think text generation is relevant to ‘fake news’.

The converse, that people who are more engaged with text generation (more feedback, more positive) can see that it is more related to fake text generation.

So, if people are educated on the topic, they will realise there is a relationship between advances in computer text simulation and fake news.

Definition of fake news

Even defining fake news is not that simple, since if you look for examples in the USA, or Syria, or anywhere else, both sides accuse the other of using it. Depends on there being no consensus as Barak Obama mentioned recently. See the bottom of the blog for some discussion.

Fake News

One of the feedback questions in the Study was about whether computer AI text generation was relevant to ‘fake news’ (false content for spam, propaganda etc.).

Q6-2: If you have heard of ‘fake news’ do you think this is relevant?

This had neutral phrasing in order not to influence replies. It did not say ‘text generator can make fake news’ as this presupposes a technical knowledge that might not be present, even after doing the experiments. Given the sometimes bizarre output of the generator, it might affect replies too specifically.

18 of the 82 respondents – 22% – replied to the question. It was the last question after a long session so perhaps some fatigue had set in. It was also a question not specifically to do with the experiment. About a quarter is still quite a high response rate.

Analysis of responses

I will be examining the actual text answers in another blog. This just looks at some basic relationships.

Several comparisons of the data were calculated and some are worth discussing here (the others are in reference material).

Comparing answer word count with positive – negative ratings:

The actual Q6 answer text was assessed for Positive – Neutral – Negative response to the ‘relevant to fake news’ question.

For instance, the text answer ‘No’ was 2 for negative, word count 1. ‘I stay out of these type debates.’ Is 1 for neutral, word count 7.

This was quite easy to do. I did not use a computer sentiment analyser as the samples were too small. Anything a bit ambiguous or with conflicting statements was rated Neutral.

Pos – neg is 0 (positive), 1 (neutral), 2 (negative) – these are scaled up so can be seen against the word count values. So low red columns are positive (yes, relevant to ‘fake news’) , high red is negative (not relevant to ‘fake news’). This is an ordinal scale.

fake news vs word count.png

Above: Pos-neg is zero for positive for ‘relevance’ to ‘fake news’. The red Pos-neg 0-1-2 value is scaled to make it easier to see. So a tall red line means negative for relevance to fake news. (There is a discussion of different statistical calculations in a forthcoming blog. The tests used in the study are Mann Whitney U, T-Tests, ANOVAS, and Pearsons, along with various bar charts and boxplots.)

At first glance this shows that higher answer word counts are associated with positive for ‘relevance to fake news’.
Low word counts are associated with negative for ‘relevance to fake news’.
There is only one negative out of the top half of the responses (1-9 on the chart above, 2-8 are pos=0).
The highest word count (column 1 on chart) was from an answer that discussed an actual example of real ‘fake news’, or propaganda in the Syrian War, so was a longer than usual negative response. It could be an outlier. After ruminating they decided it was too hard for a text generator to do, writing ‘news of any kind has to be created by humans…’ which is a value judgement (humans are best).

Statistics

Mann Whitney U test

The z-score is 4.25539. The p-value is < .00001. The result is significant at p < .05.

  • People who used the least words thought text generation was less relevant to ‘fake news’.
  • People who used more words in their answer thought it was more relevant to ‘fake news’.

Discussion

In other work, computer Sentiment Analysis revealed that all feedback answers had high scores for ‘Tentative’ and very low (no score) for ‘Confidence’. These results could be because of a general lack of experience of text generation (only 11% had previous experience).
There is a forthcoming blog on the Sentiment Analysis, there is already a blog on Sentiment Emotions (Joy etc.).

When making a positive claim (‘there is relevance’) with no ‘Confidence’ and feeling very ‘Tentative’, discursive answers are to be expected.

In the positive answers, the relevance of ‘fake news’ produced more discussion (‘Yes, and here’s why…’). This could be because the respondent felt it was relevant, but had little or no confidence in their feelings and supplied a tentative, more wordy, answer.

Whereas with a negative response, text generation is not relevant to ‘fake news’, and can be easily dismissed (‘No’ – 1 word).

Ranked Likert scores over the Study vs Pos-neg

Fake-ranked-Likert-Pos-neg.png
Mann-Whitney U Test
The z-score is 5.10963. The p-value is < .00001. The result is significant at p < .05.

This is not visually very clear until you sum the raw (before scaling for graph) scores from columns 1-8 (6) and 2-18 (10).

  • This shows that lower Likert scores (which means positive response to experiments and later questions) relates to lower Pos-neg scores (which means positive relevance to ‘fake news’). This is what would be expected.
  • People who have a lower general opinion of text generation, don’t think text generation is relevant to ‘fake news’ (and possibly anything else of practical use).

Amateur – Professional

People were asked to select Amateur/Professional (or both, scored as Pro) next to Occupations. Most people had more than one occupation (Poet and Journalist etc.) so it is not a fixed job allocation, but provides an indication that the writer either is or would like to be paid for at least some of their work.

This did not provide any significance at the ‘fake news’; response level.

There is a forthcoming blog on Amateur/Professional differences.

Comments

Is there a connection between text generation and ‘fake news’?

1/ Human pride: anthropocentrism: people that dismiss the use of text generation and its relevance to ‘fake news’, say ‘news of any kind has to be created by humans…’.

2/ The idea that a computer could not be used to write news, ‘fake’ or otherwise, might be due to the rather random output of the mid-level (GPT-2) text generator used in the Study. People might think it is producing readable nonsense, and brusquely dismiss any practical uses. This is shown in the relation between overall negative scores (Likerts) and low relevance for ‘fake news’.

3/ Perhaps dismissal of a text generation tool in regard to ‘fake news’ is indicative of not believing there is ‘fake news’ (or that it is not a problem), shown in the low word counts of their short negative responses. Further research required.

Notes

1/ Answers

If you have heard of ‘fake news’ do you think this is relevant?

Replies went from the four shortest (1 or 2 words)

If you have heard of ‘fake news’ do you think this is relevant?

No (scored 2, neg)

No! (scored 2, neg)

Sure (scored 0, pos)

Not really (scored 2, neg)

to the four longest (89, 85, 76, 50 words; 2,2,0,0)

If you have heard of ‘fake news’ do you think this is relevant?

“Unless the AI used to generate the text is extremely advanced, I think any news text generated wouldn’t be very effective — fake or otherwise. I think, at this point in our technology, news of any kind has to be created by humans to effectively educate, manipulate, or make humans react like chickens with their [thinking] heads cut off. If a text generator is going to be effective at writing propaganda, it’s skills are going to have to increase a heck of a lot. Ask again in five years.” (scored 2, neg)

“Very difficult to unpack – particularly as the different sides each claim the other side is distributing fake news. Having followed the Syrian War closely, I know for a fact our major news channels spouted fake news. Could something like this be used in social media to reply with biased opinions? Perhaps, but the existing troll farms used by all sides are often quite sophisticated, with the least sophisticated being the least effective. So, this type of low level autogeneration would not be very effective.” (scored 2, neg)

“Oh yes! Too many people are not analyzing the sources. Maybe that is too big a job or most and maybe many do not know to do this. My son repeats this claptrap to me as if it is gospel and he does not understand me when I tell him that his source is untrustworthy. He thinks if he hears something from dozens of people it must be true. Mass media is powerful in that way. “ (scored 0, pos)

“It is an enabler of fake news (eg allowing much faster fake news to be created – especially if fed in the analytics re effectiveness of dissemination etc.) but could also make REAL news reporting more efficient and effective especially if it supports optimum use of human vs AI input.” (scored 0, pos)

The text samples were too small to use computer sentiment analysis, so I scored the replies myself (human accuracy is still the best for nuanced material). Much training data for sentiment analysis is annotated by humans before use. See references.

Positive – is relevant – 0

Neutral – don’t know, don’t care or not sure – 1

Negative – is not relevant (not good enough, unsuitable, not required) – 2

How done

Sum Likerts (inverting the sixth Q (Q3/of 6 at end), the third after the 3 experiments), rank, then compare to texts, which are scored

0 pos (relevant to fake news), 1 neutral, 2 neg (not relevant to fake news).

Note: Q3/6: Do you feel that you have used somebody else’s work?

This Likert scored the other way, ie, the first option (score 1) was the most negative (is somebody else’s, ie plagiarism).

So in the calculations this score was inverted (1 become 5 etc.).

Other Calculations

1/
Overall Likert score (low positive affect) vs Fake Pos-neg relevant – neutral – not relevant

fake news vs likert.png

There is no significance to this relationship.

2/

vs. word count vs Fake news Pos-neg relevant – neutral – not relevant

fake news vs word count.png

References

Sentiment analysis papers:

https://www.kdnuggets.com/2020/06/5-essential-papers-sentiment-analysis.html

What is fake news?

Important researcher: Xinyi Zhou
These are the most recent papers on fake news:

A Survey of Fake News:

https://dl.acm.org/doi/10.1145/3395046

Fake News Early Detection: 

https://dl.acm.org/doi/10.1145/3377478

Generally, fake news is false or distorted information used as propaganda for immediate or long-term political gain. This definition separates it from advertising, which has similar approaches but is for brand promotion rather than life or death matters. False can be anything from an opinion to a website which appears to be proper news but is actually run to spread false information.

This has led to ‘reality checks’ where claims are checked against reality (which still exists) but the problem is that fake news spreads very quickly (because it appeals to the emotions, often fear or hate) while corrections take time to process and are very dull, so hardly anyone reads them, as they have no emotional content. Certainly the people that react to the fake news (if it is proven so) have moved on already, to the next item.

 

 

 

 

 

 

Writing occupation and emotions in text generation

In August 2020 research (UAL, see credits) I examined what would happen if and when writers use a computer text generator to write articles, giving them only an image prompt. The idea was to only use professional or serious amateur writers.

Go to Index of AI research

Joy, Fear, Anger, Sadness – emotion charts are after this introduction.

Can text generation help the human writing process? What do actual writers (the study respondents) think of it all?

The research examines creative and ethical concerns around the use of advanced systems, and how they will (or already do) affect stakeholders, both professional writers and serious amateurs.
Here’s the prompt image:

Prompt image man and dog
Prompt image man and dog

The results are in but I am still writing it up. So I am now dropping a few things on this blog. These are not the final results as many qualifiers need to be added, statistical definitions, significance, etc. There are over 50 charts, which is why the report is taking a long time.

More about boxplots: This is a blog about some study results boxplots. if you are not sure what it all means, please look at this first.

One question asked was whether they’d used a text generator before, someone replied ‘my unconscious’. 89% had never used a text generator before.

82 respondents from my own creativity writing app list (see below), and various professional bodies.

These are Occupation (type of writer eg, Student, Poet, Journalist etc. – see the left axis);
plotted against amount of Emotion (joy, anger etc.) in their written feedback to all the questions (summed, then scored using a sentiment analyser). (Amateur and Professional are not attached to the actual occupation, so they are on here too.)

Increased emotion values towards the right side of the chart. These plots show ranges so they only give a general visualisation.

Joy

So in the boxplot below, the most joy in responses came from Copywriters.

Perhaps they see a fantastic tool to very quickly make more copy.

Joy vs Occupation
Joy vs Occupation

Fear

The most fear in responses came from Poets and Fiction writers. Perhaps fear of losing their respect as creators of strange new worlds were no one has gone before. Or they see a fantastic tool to very quickly make them unemployed. Other and Scribbler also score on this emotion.

Fear vs Occupation
Fear vs Occupation

Anger

Would appear that Others and Scribblers are somewhat angry about something or other. More research needed! Poet and Fiction also score highly, one each here (a line).

Anger vs Occupation
Anger vs Occupation

Sadness

Perhaps poets know more sad words.

Sadness vs Occupation
Sadness vs Occupation

There’s lots more charts but that will do for today. The actual stats with significance, etc., are for future viewing.

One of the simple charts:
Time Average on Study by Occupation

Graph- Time Rank Occupations
Graph- Time Rank Occupations

Game writers had 2 outliers, one person was on it for hours. Perhaps text generation is familiar to games content writers as some games have generated scenarios. Or they have a lot of spare time – to play games.

(Possibly) confirms rumour that songs are written quickly, and that lyricists and poets have flashes of inspiration quickly recorded (and so do copywriters and scientists). Or they were in a hurry to get away…
Game and Songs, Lyrics were added by people within Other definition.

Next blog – the text generation itself.
In the experiment, people were advised to use the generator to make completed works. Several people put my name in the generator, so I became the protagonist in the stories. What!

Such as this Fiction entry:
“It was nice to hear from Geoff again. He is a reminder that life is like an ant’s journey on a blade of grass across a puddle. There is no other side to reach, because the ant is surrounded on all sides. Like an ant, like all of us, Geoff has strategies for paddling. One admires only the paddling, and not especially the termination of the journey. And perhaps that’s what should be the focus of our lives: the paddling. Not journey, not the conclusion, but the sheer determination of the paddling. With a surfer, this analogy would not work, but thinking about it, ants can’t surf.”


People used the OpenAI GPT-2 text generator in a two panel design. I’m releasing this setup as a free AI text editor soon. The generator version is Text Synth by Fabrice Bellard, who is very helpful.

University of the Arts London: my tutor at UAL CCI is Professor Mick Grierson. See Credits (new window). My app is Notes Story Board, an image and text zooming canvas.

Computer-Human Hybrid AI Writing and Creative Ethics

Introduction

This blog is about my 2020 research into computer text generation and the effects on professional ands amateur writers. I am working on this topic at the University of the Arts London (UAL CCI, Dir. Mick Grierson).

No-one has asked creatives or writers what they think of the new ‘AI’ systems that generate readable text and so directly threaten their jobs, and could change the way people work forever (or don’t work forever). This is a topic that directly impinges on self-worth and financial worth in more ways than anyone can imagine, although plenty are worrying.

STUDY – ONLINE EXPERIMENT
August-October 2020

I devised an online experiment about this topic, allowing respondents to experiment with creating hybrid stories using a text generator. The people were all professional or serious amateurs (and a couple of small students) invited from my own creative writing software mailing list, a couple of writing forums, and a publisher’s writers’ forum, plus friends and relatives who generally use writing in their work. Credits are at the bottom.

Text generation

You might have heard of Google OpenAI’s GPT-2 and GPT-3. My experiment uses a generating system (Fabrice Bellard’s Text Synth, with permission)  based on GPT-2, that anyone can use. GPT-2 was used here as the model works well for idea generation and is more generally available at the time than GPT-3, which is much larger.

Note: The text generation and editing system is now a free online tool (creativity support tool or CST) at

Story Live writing with AI free online

The experimental results will feed into this blog (see Index for different aspects) and later an academic paper, and also a new book for the general public on the whole subject of computers, creativity and writing.

Please sign up for news and notifications – there’s a form on this page.

Brief description of the Study

Below is a graphic of the entire online study. Each block is a page and journey was left to right from top to bottom. The three text generation and editing experiments used a similar set up to the Story Live tool.

Each writing experiment – Caption, News and Fiction – had a question afterwards, then there were more questions after the experiments (see diagram below). All this will be addressed in blogs here, along with other discussions.

The image writing prompt was the same for each experiment and for all respondents for uniformity (there is a blog on the man and dog here).

Prompt image man and dog
Prompt image man and dog
Flowchart of Study

Geoff Davis

The computer support tool (CST) from this study is Story Live writing with AI free online

My other creativity tools are Notes Story Board and Story Lite from my Story Software. For my other activities please see the home page of this site.

Study

This study was devised and the site programmed by Geoff Davis for post-graduate research at University of London Creative Computing Institute UAL CCI 2020. The Supervisor is Professor Mick Grierson, Research Leader, UAL Creative Computing Institute.

Text Synth

Text Synth, by Fabrice Bellard, is a publicly available text generator, was used as this is the sort of system people might use outside of the study. It was also not practical to recreate (program, train, fine-tune, host) a large scale text generation system for this usability pre-study. Permission was granted to use Text Synth in the study by Fabrice Bellard Jul 7 2020.

Fabrice Bellard, coder of Text Synth.
Fabrice is an all-round genius and writes a lot of OS. Text Synth was built using the GPT-2 language model released by Google OpenAI. It is a neural network of 1.5 billion parameters based on the Transformer architecture.