Discussion of Ethiopian OLPC Study - "Write" is Favorite Activity

In a previous post, I mentioned a study done  by the University of Groningen of a small Ethiopian OLPC project, published in 2009, called  Does Technology Drive Social Change? 

One finding the study reported was that the children's favorite activity, by far, was Write. Other favorite activities that scored much lower (3-13%) were Record, Paint, Game, Calculate, Chat, and Memory. The study's authors concluded that, because writing was the most favored activity, the laptops are "a learning device!"  (exclamation point theirs).

The usefulness of this data is limited because the study only reports "favorite activity," not the amount of time each activity was actually used. It is hard to tease meaningful information out of this.

Keeping in mind the limited information we have, I draw a different conclusion.  To me, their finding says that the laptops are being used as glorified pencil and paper. 

Granted, writing is a great educational activity, but laptops aren't necessary for children to write.  There are much less expensive, even free, options available for that.  In Lesotho, writing is a skill that students master in the traditional classroom without computers.

What I want to see in the results from studies like this is a much higher percentage of other XO laptop activities being used, activities that aren't otherwise available and which develop critical thinking and problem-solving skills. 

I'd especially like to see children say their favorite activity is something otherwise not available to them, like:

• Browse - which means the children are searching out information,
• eToys, Turtle Art, and Scratch - which develop advanced thought processes,
• Write used in conjunction with other activities like Paint and Record - which means the children are performing more complex tasks,
• Tam Tam - which stimulates creativity and abstract thinking,
• TuxMath - because math is a skill not adequately acquired in the standard classroom (in Lesotho anyway), and
• even games like Maze and Implode - which require conceptualization and planning.

One of L2L's primary objectives is to teach children advanced cognitive skills.  That is the key advantage we see in using the XO laptops. 

I can hardly wait to see what we find out about our project after we conduct our own evaluation.  Early on, I'm guessing we will see results very similar to the Ethiopian study, except maybe with Record getting a higher percentage.  Hopefully though, as time goes by and we continue to work with teachers and students, we will see more complex activities like Scratch (my favorite) score the highest.  When that happens, I'll know we have succeeded.

-- Janissa



 

Volunteer Needed to Design Evaluation Methodologies

We're looking for a volunteer evaluation consultant. 

We need to select and customize survey techniques to evaluate our project in Lesotho.  In the short-term, we want to look at changes in academic achievement and computer skills and evaluate supplemental materials used with the school curriculum.  (We've got an independent researcher working on evaluating long-term educational, social, economic, health, and cultural changes.)

Is there anyone out there who has knowledge and experience designing evaluation methodologies who would like to work with Laptops to Lesotho between now and January? 



Nohana Primary School, Laptops to Lesotho

We'd like to start collecting some data in January when we have a group of volunteers going to Lesotho for several weeks.  We've defined our evaualtion goals and the basic data we want to collect.  Now we need to get down to the nitty-gritty and develop detailed methodologies.

Can you help or know someone who could?

 

-- Janissa

 

 

 



Evaluating Impact & Effectiveness - Ethiopian Study

We are hoping to start collecting data for a quantitative evaluation of our project's impact and effectiveness in January 2013.  We will be looking at both short-term and long-term changes.

 

There was a study done of a small Ethiopian OLPC project by the University of Groningen, published in 2009.  It is called Does Technology Drive Social Change?.  (See the report at http://www.gg.rhul.ac.uk/ict4d/NinaandTom.pdf.) 

 

This report is not a rigorous presentation, but it does summarize general results of the study.  They compared 3 schools with XO laptops to 3 without.  They found:

• Most of the children used their laptops everyday at home and often at school. 
• They used them mostly to learn, much less to play or communicate.
• Their favorite activity, by far, was writing.
• They sometimes shared them with their parents and friends.
• After two years, social and educational changes were seen in the locations with laptops but only personal growth was seen in places without them.

Other results I found particularly interesting, given the remote rural location of our work, were:

• In the rural school, there was a slight increase in valuing the equal treatment of boys and girls. This was not seen in the city schools, which had much higher values for this even before the introduction of computers.
• The same was true for attendance.  In the rural school, the laptops greatly improved motivation to attend school, whereas city schools didn't see much change because motivation was already fairly high.

I like that this study used a control group that didn't have laptops for comparison.  We consider that a very important component of our evaluation.  I'd also like to see our project do much more in-depth analysis of impacts than they reported in the Ethiopian study.

See related L2L blog posts: Project Evaluation Meeting published on 11 July 2012 and Inquiry into Scientific Evaluation of L2L, with replies, published on 13 May 2011.

- Janissa

 

Looking for Funding for a Project Evaluation

Does anybody know of a foundation or organization(s) that might be interested in funding a rigorous, in-depth quantitative evaluation of our project's short-term and long-term impacts and the project's effectiveness?  We have an interested researcher, but we need funding. 

We think it's important to evaluate our work because we are using a different approach than One Laptop Per Child (OLPC) and most other computer projects in developing nations.  (See our 13 May 2011 post for a summary of our approach.)

- Janissa

Project Evaluation Meeting

On Monday 09 July 2012, Janissa Balcomb, Fortunate Gunzo, and Sherrie Howey held a conference call via Skype to discuss the project evaluation.  Janissa will be sending out a draft brief this week to interested parties for comments and suggestions. 

Why an evaluation is important

• We want to determine if the approach we are using is effective and if there are ways we can improve it.
• We need to determine if the project is successfully meeting our objectives or not.

Short-term aims: to evaluate

1. project implementation process
2. student academic achievement
3. student computer-related job skills
4. role of the computer program within the school curriculum
5. views about the project

Data for short-term aims 2-4 will be collected by L2L volunteers.  Fortunate and Sherrie will be brainstorming methodology for these aims and will meet again in August to draw up details. 
Data for aims 1 and 5 will be collected by independent outside people.

Long-term aims: to evaluate

1. long-term impacts on students, including
   - education (highest level achieved)
   - employment (income level, job type and level),
   - finances (annual income, net worth, debt)  
   - societal (status in the community, size and quality of social network),
   - family (marital status, number of children), and
   - health (psychological stability, risky behaviors , physical condition, and longevity).
2. long-term impact on teachers, including- skills acquired or improved on, such as computer proficiency, teaching skills, language and math skills, and interpersonal communications, 
   

   - formal education (highest level achieved), 

   - employment (income level, job type and level, location of employment),
   

   - finances (annual income, net worth, debt)  

   - societal (status in the community, size and quality of social network), 
   - family (marital status, number of children), and
   

   - health (psychological stability, risky behaviors , physical condition, and longevity

Data for both long-term aims 1 and 2 will be collected by independent outside people.

We hope to have 1-2 control groups for comparison, one at a school with no computers and possibly another school with computers that were deployed using a more traditional approach (i.e. computers are deployed with little or no preparation of the human resource prior to deployment, training and support is minimal or short-lived, and no effort is made to provide supporting educational materials or to incorporate the computers into the existing curriculum).

Janissa, Fortunate, and Sherrie will meet in September to finalize the evaluation plan. We hope to begin implementing the evaluation process and collecting data starting in January 2013.

Inquiry into Scientific Evaluation of L2L // Replies Received

[Email sent by Janissa to Research@lists.laptop.org an OLPC-za@lists.laptop.org , Replies follow.]

We are using what we think is a rather unique approach to setting up a computer program in a developing nation, and we would like to find a research partner to do an in-depth, long-term evaluation of the efficacy of our project. 
The first thing we did differently was we took 1½ years to establish a grassroots organization in the local community before we distributed a large number of computers. We started with just two Windows-based computers and two XO-1 laptops. With the help of a Peace Corps Volunteer living in the community, we put two local educators in charge of the project at the very onset. We mentored them, trained them, and made them responsible for all major decisions. From the onset, we let them know that our role was merely as facilitator to get them started and that, within a matter of a few years, they would be solely responsible for the project.

As we guided these two project leaders, we had a series of bench marks, unbeknownst to them, that they had to achieve before we moved to the next step. It wasn't until they reached the point where they had enough computer skills to supervise the project, where they were communicating regularly with us by email, where they had shown a serious sustained commitment to the project, and where the community had shown full investment in the project that we began delivering laptops.

Another thing we are doing is moving at a pace of change that the community can fully absorb without disruption. Our first deployment in 2010 consisted of 50 laptops. Our second deployment with a similar number of laptops will be a full year later.

During the first deployment, we met separately with all the teachers, parents/guardians, students, community leaders, religious leaders, government officials, and local police. After those meetings, we spent ten days helping the project leaders and school staff work out rules and regulations to govern the project. These rules try to address every possible scenario we could foresee and establish a procedure to deal with situations we couldn't foresee. From this, the school staff developed contracts for each student, parent/guardian, and educator to sign in order to participate in the project.

The regulations and contracts clearly define responsibilities, benefits, and penalties for all parties involved. Penalties for violating the contract can be paid in cash or worked off by doing community service. (The English version of the Rules and Regulations, Contracts, and Fine Schedules is posted on our blog at http://olpc2010-lesotho.blogspot.com/p/2011-rules-regulations-contracts-fee.html)

Originally, we based our project on the OLPC philosophy and guidelines. However, the leaders, educators, and community members felt that one aspect, child ownership, didn't fit well with their situation and the number of laptops we were providing. They changed that to school ownership with a system that allows the children and teachers to check the laptops out like a library book.

As part of this system, a student must earn the right to check out a laptop. First, both the student and their parents/guardian must all sign contracts agreeing to abide by the Rules and Regulations. Then, the student and parents/guardian must learn how to properly care for the laptop and display that knowledge to the satisfaction of the student's classroom teacher. Lastly, the student must earn a set number of points, via a clearly defined point system, based on the student's behavior at school and at home.

During the first deployment we spent three weeks at the end of the school year giving all the teachers at the school and a principal from another school in the area intensive training on how to use the XO laptop, how to charge and repair the laptops, how to teach with laptops in the classroom, and how to develop lessons with the laptops to supplement the curriculum.

Then we left. Three months later, one of the project leaders, who is the school principal, was brought to the U.S. for a professional and cultural exchange. During that time, he had the opportunity to visit a number of schools, observe classes, and talk to principals, teachers, students, and school board members to learn about the U.S. education system. (The trip was paid for by FIPE, the Foundation for International Professional Exchange.) He returned to his school with a new perspective and new ideas.

Laptops to Lesotho volunteers will return to the village in December 2011. In the interim, the school staff is running the program. So far, the laptops are being used several times a week in grades 4-7 and periodically in the lower grades. Most of the teachers are using them in their classrooms, though two are not yet comfortable teaching with them. In order that all the students get a chance to use the laptops, those teachers switch with other teachers for some lessons. A larger solar power system is being installed this month that will enable the teachers to charge more laptops at one time and that will allow them to use the laptops even more frequently in the classroom.

It is too early to tell what significant long-term changes this project will make, but in the short-term it has been very successful. Some of the short-term changes we have documented at this early stage include a 20% increase in enrollment at the school and a drop in chronic absenteeism to nearly zero. Empirical evidence shows a vast improvement in student behavior and an increased rate of improvement in math and English skills.

The project has also gotten the parents/guardians more actively involved in the school and has brought the community more closely together. They have decided to start an annual cultural celebration, a tradition that had been lost prior to this project. They will use the celebration, along with other community activities, to help raise funds for the project.

The school staff selected two dozen students from grades 5-7 to serve as student leaders for the project. The staff has already begun to evaluate schools and school staff members in surrounding villages to determine the next candidate for expansion of the project. They have also decided to spend part of the project funds to attend a grant-writing and fundraising workshop. 
We think that our process could be replicated successfully elsewhere. But, before we get too much farther down the road, we would like to establish a strict scientifically-based hypothesis testing research project to evaluate this technique both in short-term and long-term gains.

If you are interested in helping to set up this evaluation research, or know of someone who might be, please let me know.

--------------------------------------------------------------------------------

----- Original Message -----
From: Stacey Kertsman
To: Temina Madon
Cc: Janissa Balcomb; Wayne Getz
Sent: Wednesday, May 11, 2011 1:26 PM
Subject: Fwd: [Research] new subscriber intro - looking for partner to evaluate our program

Temina,
Just thought I'd forward this along! :) Maybe some student at UCB will be interested.....

Stacey

---------------------------------------------------------------
---- Original Message ----
Subject: Re: Fwd: [Research] new subscriber intro - looking for partner to evaluate our program
From: Janissa Balcomb
Date: Wed, May 11, 2011 3:04 pm
To:      Stacey Kertsman

Hi Stacey,

Thanks for forwarding this!

Janissa
 ___________________________________________

----- Original Message -----
From: Edward Cherlin
To: Janissa Balcomb
Cc: research@lists.laptop.org
Sent: Wednesday, May 11, 2011 9:19 PM
Subject: Re: Re: [Research] new subscriber intro - looking for partner to evaluate our program

On Wed, May 11, 2011 at 23:07, Janissa wrote:
Janissa:  Edward, Thanks for your input.

Edward: A pleasure.

Janissa: My replies follow
 __________________

Edward: "I would also suggest that you use the Free Software/Open Source approach of publishing all of your materials and letting the community work with them, try to improve them, test them, and so on. The new, in testing, not quite yet public Replacing Textbooks server at Sugar Labs is available for hosting training materials in addition to Free digital > textbook replacements, or Open Education Resources (OER)."

Janissa: We plan to make all our materials public. Anything we create > here in the U.S. for the school will be posted; however, it is very difficult for the teachers in Lesotho to post materials they have created > because there is no affordable internet service available in the area. > When we visit the school later this year, we will make copies of anything they have created and will post it for them.

Edward: I wasn't sufficiently explicit. I mean not only public, but under a Creative Commons or similar license, preferably permitting reuse, improvement, translation, and republication with credit but without having to get explicit permission.

Sneakernet (originally with floppy disks) is an ancient and venerable tradition in the computer community.

Janissa: Will the OER/free digital texbooks be provided in a format that can be downloaded on a flash drive rather than used interactively online?

Edward:  It is our intention to make materials that can be used on an XO, and in Sugar on any other platform, and in many cases that means PDFs and other public document formats that can be used anywhere on anything.
__________________ 

Edward: "Are you familiar with Sarvodaya in Sri Lanka? Your methods are somewhat similar.  Sarvodaya Shramadana Movement http://www.sarvodaya.org/"

Janissa: I am not familiar with Sarvodaya, but I will check [it] out.
 __________________

Edward: "Did you record any of those meetings? Such recordings would be of inestimable value to researchers and to other instructional designers."

Janissa: No, we did not record the meetings other than just taking notes. I think having an audio or video recorder present might have inhibited the process too much. It was hard enough as it was to get the teachers to talk, express opinions, and make decisions. There was A LOT of cajoling involved.

Edward: Understood. Perhaps they will at some point become comfortable with students recording them using XOs, or they can record each other. 
__________________

 Edward: "Is your solar power system design public?" 

Janissa: It was designed by the Bethel Community Development and Business Center in Lesotho. I will check with them about making the design public.

Edward:  Thank you. Illinois Institute of Technology is doing another design for schools with XOs in Haiti, and there are others. I encourage organizations doing this to get together and share expertise.
__________________ 

Edward: "I would be very interested to see a comparison of your results with a one-to-one deployment. There are many other experiment designs of interest." 

Janissa: We had originally hoped to have a one-to-one ratio, but financial constraints prevented that. We thought we would still work > toward that goal eventually, but the community has over-ridden us. They would prefer that we expand to other schools rather than try to reach a one-to-one ratio at the school we are currently working with. The situation could definitely make for some interesting comparisons, if the research is set up properly.

Edward: It would require some care to control for a variety of variables. We can discuss that when we get some experiment designers involved. 
__________________

Edward: "I don't have the resources of a researcher, and Sugar Labs is not a research group. But as a Sugar Labs Project Manager, I would be very interested in following your research, and suggesting some research directions."

Janissa: I would love to get any suggestions you or others have. I'm completely new to this role and am feeling my way as I go, so any help is greatly appreciated.

Edward: I know some people we should talk to. I'll ask some of them individually. To begin with, we should talk to the Sugar Labs It's-An-Education-Project mailing list. I have copied them on this. --

Edward Mokurai (默雷/धर्ममेघशब्दगर्ज/دھرممیگھشبدگر ج) Cherlin
Silent Thunder is my name, and Children are my nation. The Cosmos is my dwelling place, the Truth my destination. http://www.earthtreasury.org/

--------------------------------------------------------------------------------

----- Original Message -----
From: Arthur Attwell
To: Janissa Balcomb
Sent: Thursday, May 12, 2011 1:38 AM
Subject: Re: [Olpc-za] looking for someone to do scientific evaluation of OLPC-like project

Hi Janissa

This is fascinating, thank you. Is this info online as a post (I didn't see it on the OLPC Lesotho blog)? I'd love to tweet it at [at] electricbook.

Best wishes
Arthur

Arthur Attwell
Chief Executive, EBW

--------------------------------------------------------------

----- Original Message -----
From: Janissa Balcomb
To: Arthur Attwell
Sent: Thursday, May 12, 2011 10:51 AM
Subject: Re: [Olpc-za] looking for someone to do scientific evaluation of OLPC-like project

Hi Arthur,

Feel free to tweet or pass this along any which way. I'm going to post it to our blog today.

Janissa

--------------------------------------------------------------------------------

----- Original Message -----
From: Eckart Zollner
To: Janissa Balcomb
Sent: Thursday, May 12, 2011 3:40 AM
Subject: RE: [Olpc-za] looking for someone to do scientific evaluation of OLPC-like project

Hi Janissa,

I have done quite a bit of work in this sector and would be interested to assist in this research project.

Kind regards

Eckart Zollner

Implementation of "Practical advice that works"

-----------------------------------------------------

----- Original Message -----
From:Janissa Balcomb
To: Eckart Zollner
Cc: Fortunate Gunzo ;
Sent: Thursday, May 12, 2011 11:16 AM
Subject: Re: [Olpc-za] looking for someone to do scientific evaluation of OLPC-like project

Hi Eckart,

We have a team member in South Africa at Rhodes University who is overseeing this. Her name is Fortunate Gunzo and her email address is . She is putting together a brief for the research. Please feel free to contact her to discuss this. If you could copy me on any emails with her, I'd appreciate it.

Thanks,
Janissa

--------------------------------------------------------------------------------

----- Original Message -----
From: Henk Boshoff
To: Janissa Balcomb ;
Sent: Thursday, May 12, 2011 4:14 AM
Subject: Research project.

Hi Janissa,

I would like to introduce myself/my business, as I am interested in performing the research project. I am an engineer with a masters degree from the University of Pretoria, South-Africa. Main subjects include systems engineering, engineering management and I have done some research to do a PhD in Technology management, which was put on hold when I started my business S-Curve Technologies cc. www.s-curve.co.za I have attached a more formal description of my business to this mail. Some of my published work may be viewed at:

http://www.evancarmichael.com/African-Accounts/1683/summary.php

http://free.financialmail.co.za/innovations/1oct/dinn.htm

My business has a strong focus on data communications design, and I have a personal passion for NGO work. The following video indicates the work performed by AED Sattelife, for which I designed a data routing system. http://www.youtube.com/watch?v=f_irCIDH3JI

I would appreciate further communications regarding this research,

Regards
Henk Boshoff.

Henk Boshoff (Founder) S-Curve Technologies cc.

-------------------------------------------------

----- Original Message -----
From: Janissa Balcomb
To: Henk Boshoff
Cc: Janissa ;
Sent: Thursday, May 12, 2011 11:08 AM
Subject: Re: Research project.

Hi Henk,

Thanks for the information. We have a team member in South Africa at Rhodes University who is overseeing this. Her name is Fortunate Gunzo, and her email address is . She's putting together a brief for the research. Please feel free to contact her to discuss this, and if you would, cc me.

Thanks,
Janissa


--------------------------------------------------------------------------------

On 12 May 2011 03:42, Valerie Taylor wrote:

Hi Phil

This came in a post to the One Laptop Per Child group. How are they doing in empowering the community, based on information provided?

All the best, ..
Valerie

------------------------------------------

----- Original Message -----
From: Phil Bartle
To: Valerie Taylor
Cc: Jonathon Tardif
Sent: Thursday, May 12, 2011 4:51 AM
Subject: Re: Community empowerment

Thanks, Valerie
From what I read in their description they are going about it in a right way (not that there is only ONE right way). I would have a few questions, but no surprise they are not answered in the blog.

Warm Regards,
Phil

Community Empowerment: www.scn.org/cmp/

-------------------------------------------

On Thu, May 12, 2011 at 12:27 PM,
Janissa Balcomb wrote:

 Hi Phil & Valerie,

I welcome your input and suggestions for our Laptops to Lesotho project.  I'm completely new at this; my background is not in community organization or even anything close.

What questions didn't I answer? We are a very small, all-volunteer group,
and I confess that I have to wear many unfamiliar hats while trying to
juggle several other jobs. I often don't have time to keep the blog current
with full details. And because of poor connectivity, information from the
village is often delayed. But, I will try to answer any questions you have.

It's kind of a long story how I got into this, but essentially the community
contacted me and asked for help. I'm feeling my way as I go, so I
appreciate all the help I can get.

Thanks,
Janissa

-------------------------------------------------

----- Original Message -----
From: Valerie Taylor
To: Janissa Balcomb
Cc: Phil Bartle ; Jonathon Tardif
Sent: Thursday, May 12, 2011 12:13 PM
Subject: Re: Community empowerment

Hi Janissa

Phil and a group of wonderful volunteers have put together a users
guide for community empowerment.  http://www.scn.org/cmp/

There are many good ideas presented in a very accessible manner. And
if English is not the first language for community facilitators,
translations into other languages are available as well.

We really appreciated that you shared your work and look forward to
hearing more.

All the best
..Valerie

-----------------------------------------------------

----- Original Message -----
From: Janissa Balcomb
To: Valerie Taylor
Sent: Thursday, May 12, 2011 12:26 PM
Subject: Re: Community empowerment


Thanks! I will check it out. Janissa

--------------------------------------------------------------------------------
----- Original Message -----
From: Sarah Howard
To: Janissa Balcomb
Sent: Thursday, May 12, 2011 5:54 PM
Subject: laptops to Lesotho

Hi Janissa,

I just read your post in the OLPC digest. I thought I could be of some help. My name is Dr Sarah Howard and I'm a lecturer at the University of Wollongong in Australia, formerly an high school art teacher from San Francisco. You can find more about me at these two links:

1) Academia.edu: http://uow.academia.edu/SarahHoward/About
2) UOW faculty page: http://tinyurl.com/6kj8tum

You'll see in my bios that I run a three-year (2010-2012) state-wide one-to-one laptop evaluation in New South Wales. It's one of the largest ever conducted, on this kind of program. I also work with researchers at the University of Cape Town and Rhodes University in Grahamstown, looking at ICT and laptop use in South African schools. I have an particular interest in the Teacher Laptop Initiative roll-out in South Africa. I was over there in November 2010 doing a data collection, currently analyzing the results. I am also working with OLPC Australia on their first program evaluation.

All this said, I'm very interested to expand my work in South Africa and continue to look at laptop roll-out strategies in different contexts. Possibly we could skype (sahoward.uow) to talk a bit more about the work you are doing? I leave for France and the UK, for research, in a few weeks. If you are interested in chatting, it would be good to talk before I leave on May 29th.

I look forward to hearing more about your program.

Thanks so much,
Sarah

Dr Sarah K. Howard
Lecturer in Education
ICT Undergraduate Program Coordinator
Faculty of Education, 67.309
University of Wollongong
Wollongong, 2522 NSW
Australia

----------------------------------------

----- Original Message -----

From: Janissa Balcomb

To: Sarah Howard

Cc: Fortunate  Gunzo 

Sent: Thursday, May 12, 2011 6:34 PM

Subject: Re: laptops to Lesotho

Hi Sarah,

Thanks for your interest in our program. I've forwarded your email to a member of our team, Fortunate Gunzo, who is overseeing the evaluation process for us. Maybe you know her? She is at Rhodes University in Grahamstown. Her email address is          .

Thanks again,

Janissa

------------------------------------------------