An N-of-1 Evaluation Framework for Behaviour Change Applications

Claire McCallum; John Rooksby; Parvin Asadzadeh Birjandi; Cindy M. Gray; Matthew Chalmers

doi:10.1145/3290607.3312923

An N-of-1 Evaluation Framework for Behaviour Change Applications

Claire McCallum, John Rooksby, Parvin Asadzadeh Birjandi, Cindy M. Gray, Matthew Chalmers

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Citations (Scopus)

57 Downloads (Pure)

Abstract

Mobile behaviour change applications should be evaluated for their effectiveness in promoting the intended behavior changes. In this paper we argue that the 'gold standard' form of effectiveness evaluation, the randomised controlled trial, has shortcomings when applied to mobile applications. We propose that N-of-1 (also known as single case design) based approaches have advantages. There is currently a lack of guidance for researchers and developers on how to take this approach. We present a framework encompassing three phases and two related checklists for performing N-of-1 evaluations. We also present our analysis of using this framework in the development and deployment of an app that encourages people to walk more. Our key findings are that there are challenges in designing engaging apps that automate N-of-1 procedures, and that there are challenges in collecting sufficient data of good quality. Further research should address these challenges.

Original language	English
Title of host publication	CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems
Subtitle of host publication	Glasgow, Scotland Uk — May 04 - 09, 2019
Place of Publication	New York
Publisher	ACM
Number of pages	6
ISBN (Electronic)	9781450359719
DOIs	https://doi.org/10.1145/3290607.3312923
Publication status	Published - 2 May 2019
Event	ACM CHI Conference on Human Factors in Computing Systems 2019: CHI’19 Workshop: HCI in China: Research Agenda, Education Curriculum, Industry Partnership, and Communities Building - Scottish Event Campus, Glasgow, United Kingdom Duration: 4 May 2019 → 9 May 2019 https://chi2019.acm.org/ http://chi2019.acm.org

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	ACM CHI Conference on Human Factors in Computing Systems 2019
Abbreviated title	CHI 2019
Country/Territory	United Kingdom
City	Glasgow
Period	4/05/19 → 9/05/19
Internet address	https://chi2019.acm.org/ http://chi2019.acm.org

Keywords

Mobile Health
Behaviour Change
Evaluation
Single Case Design
Mobile Application

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1145/3290607.3312923

mccallumNof1_chi2019Accepted author manuscript, 215 KB

Cite this

McCallum, C., Rooksby, J., Asadzadeh Birjandi, P., Gray, C. M., & Chalmers, M. (2019). An N-of-1 Evaluation Framework for Behaviour Change Applications. In CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems: Glasgow, Scotland Uk — May 04 - 09, 2019 Article LBW1810 (Conference on Human Factors in Computing Systems - Proceedings). ACM. https://doi.org/10.1145/3290607.3312923

@inproceedings{677ce660b44248f5bdd6fa55e5e9664e,

title = "An N-of-1 Evaluation Framework for Behaviour Change Applications",

abstract = "Mobile behaviour change applications should be evaluated for their effectiveness in promoting the intended behavior changes. In this paper we argue that the 'gold standard' form of effectiveness evaluation, the randomised controlled trial, has shortcomings when applied to mobile applications. We propose that N-of-1 (also known as single case design) based approaches have advantages. There is currently a lack of guidance for researchers and developers on how to take this approach. We present a framework encompassing three phases and two related checklists for performing N-of-1 evaluations. We also present our analysis of using this framework in the development and deployment of an app that encourages people to walk more. Our key findings are that there are challenges in designing engaging apps that automate N-of-1 procedures, and that there are challenges in collecting sufficient data of good quality. Further research should address these challenges.",

keywords = "Mobile Health, Behaviour Change, Evaluation, Single Case Design, Mobile Application",

author = "Claire McCallum and John Rooksby and {Asadzadeh Birjandi}, Parvin and Gray, {Cindy M.} and Matthew Chalmers",

year = "2019",

month = may,

day = "2",

doi = "10.1145/3290607.3312923",

language = "English",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "ACM",

booktitle = "CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems",

address = "United States",

note = "ACM CHI Conference on Human Factors in Computing Systems 2019 : CHI{\textquoteright}19 Workshop: HCI in China: Research Agenda, Education Curriculum, Industry Partnership, and Communities Building, CHI 2019 ; Conference date: 04-05-2019 Through 09-05-2019",

url = "https://chi2019.acm.org/, http://chi2019.acm.org",

}

McCallum, C, Rooksby, J, Asadzadeh Birjandi, P, Gray, CM & Chalmers, M 2019, An N-of-1 Evaluation Framework for Behaviour Change Applications. in CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems: Glasgow, Scotland Uk — May 04 - 09, 2019., LBW1810, Conference on Human Factors in Computing Systems - Proceedings, ACM, New York, ACM CHI Conference on Human Factors in Computing Systems 2019, Glasgow, United Kingdom, 4/05/19. https://doi.org/10.1145/3290607.3312923

An N-of-1 Evaluation Framework for Behaviour Change Applications. / McCallum, Claire; Rooksby, John; Asadzadeh Birjandi, Parvin et al.
CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems: Glasgow, Scotland Uk — May 04 - 09, 2019. New York: ACM, 2019. LBW1810 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An N-of-1 Evaluation Framework for Behaviour Change Applications

AU - McCallum, Claire

AU - Rooksby, John

AU - Asadzadeh Birjandi, Parvin

AU - Gray, Cindy M.

AU - Chalmers, Matthew

PY - 2019/5/2

Y1 - 2019/5/2

N2 - Mobile behaviour change applications should be evaluated for their effectiveness in promoting the intended behavior changes. In this paper we argue that the 'gold standard' form of effectiveness evaluation, the randomised controlled trial, has shortcomings when applied to mobile applications. We propose that N-of-1 (also known as single case design) based approaches have advantages. There is currently a lack of guidance for researchers and developers on how to take this approach. We present a framework encompassing three phases and two related checklists for performing N-of-1 evaluations. We also present our analysis of using this framework in the development and deployment of an app that encourages people to walk more. Our key findings are that there are challenges in designing engaging apps that automate N-of-1 procedures, and that there are challenges in collecting sufficient data of good quality. Further research should address these challenges.

AB - Mobile behaviour change applications should be evaluated for their effectiveness in promoting the intended behavior changes. In this paper we argue that the 'gold standard' form of effectiveness evaluation, the randomised controlled trial, has shortcomings when applied to mobile applications. We propose that N-of-1 (also known as single case design) based approaches have advantages. There is currently a lack of guidance for researchers and developers on how to take this approach. We present a framework encompassing three phases and two related checklists for performing N-of-1 evaluations. We also present our analysis of using this framework in the development and deployment of an app that encourages people to walk more. Our key findings are that there are challenges in designing engaging apps that automate N-of-1 procedures, and that there are challenges in collecting sufficient data of good quality. Further research should address these challenges.

KW - Mobile Health

KW - Behaviour Change

KW - Evaluation

KW - Single Case Design

KW - Mobile Application

UR - http://www.scopus.com/inward/record.url?scp=85067291898&partnerID=8YFLogxK

U2 - 10.1145/3290607.3312923

DO - 10.1145/3290607.3312923

M3 - Conference contribution

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems

PB - ACM

CY - New York

T2 - ACM CHI Conference on Human Factors in Computing Systems 2019

Y2 - 4 May 2019 through 9 May 2019

ER -

McCallum C, Rooksby J, Asadzadeh Birjandi P, Gray CM, Chalmers M. An N-of-1 Evaluation Framework for Behaviour Change Applications. In CHI EA 2019 - Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems: Glasgow, Scotland Uk — May 04 - 09, 2019. New York: ACM. 2019. LBW1810. (Conference on Human Factors in Computing Systems - Proceedings). doi: 10.1145/3290607.3312923

An N-of-1 Evaluation Framework for Behaviour Change Applications

Abstract

Publication series

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this