The first step of the model is to help ensure that as much useful background information about the app is known before you evaluate it. This information helps create a useful context in which you can consider using the app and provides a framework for your decision making as you progress through the model. Thus, the questions below will help you decide whether to proceed with the app evaluation. You do not need to have an answer for each question in order to proceed with evaluating an app.

1. Does the app identify ownership?
2. Does the app identify funding sources and conflicts of interest?
3. Does the app come from a trusted source?
4. Does is claim to be medical?
5. Are there additional or hidden costs?
6. Does the app work offline?
7. On which platforms/operating systems does it work?
   • Does it work on a desktop computer?
8. Does the app work with accessibility features of the iPhone/android?
   • Is it accessible for those with impaired vision or other disabilities?
9. Has the app been updated in the last 180 days?

While nearly any measurement or intervention contains some risk (e.g., physical, psychological, legal, social, and economic), apps present some unique risks that may often be overlooked. Risks may include data costs associated with app use (i.e., depending your contracted data plan with your wireless provider), social profiling, loss of insurance benefits or insurability—all of which are associated with privacy and security. Digital privacy and security are not often high-level risk factors when prescribing a medication or conducting in-person therapy; when deciding on whether to use an app, however, they are extremely important and should be the first area evaluated.

The questions below are intended to help you consider many aspects of app security and privacy. Note that they are not all-inclusive, as there is currently no "gold standard" for rating apps' privacy and security. Many of your answers to these questions should be found in the app's privacy policy, or perhaps on the app developer's web site. If there is no privacy policy that you could readily find, you should consider whether the app is appropriate for collecting sensitive personal health information.

For certain questions, like what security measures are in place, it is necessary to take the app's description at face value at this time. There is no cut-off or score for this level of the model; instead you and the patient will need to decide if—based on the answers to these questions—you feel the app meets your standards. However. If they are not addressed or are addressed in a way suggesting that patient privacy and security may be compromised, you should consider whether this is appropriate for collecting sensitive personal health information. The ultimate goal of this level is to ensure an app will not cause harm by violating patient safety, security and privacy and that you and the patient understand the scope and limitations of privacy and security.

1. Is there a transparent privacy policy that is clear and accessible before use?
2. Does the app declare data use and purpose?
3. Does the app describe use of PHI?
   • Deidentified vs. anonymous?
4. Can you opt out of data collection or delete data?
5. Are data maintained in the device or on the web?
6. Does the app explain security systems used?
7. Does the app collect, use, and/or transmit sensitive data? If yes, does it claim to do so securely?
8. What third parties does the app share data with?
9. If appropriate, is the app equipped to respond to potential harms or safety concerns?

App developers often make many claims on their apps' clinical effectiveness or background—though there is often little data to support the claims. This does not mean that apps don't work, but rather that there is much we still do not know. If you decide that an app has sufficient privacy and security at Level 2, then your task at Level 3 is to evaluate any evidence for potential benefits.

While some apps' benefits have been documented in clinical studies, many—if not most—have not. Because of this, we recommend that you download and try the app to see what it is actually doing and if the content and information it offers appear at least reasonable and not harmful (i.e., evidence of "face validity"). Again, few apps will have a gold standard, randomized double blinded placebo controlled study to suggest they are effective, so the questions presented below are designed to help you think of other ways you can make an informed decision about an app's evidence base.

1. Does the app appear to do what it claims to do?
2. Is the app content correct, well-written, and relevant?
3. What are the relevant sources or references supporting the app use cases?
4. Is there evidence of specific benefit from academic institutions, publications, end user feedback, or research studies?
5. Is there evidence of effectiveness/efficacy?
6. Was there an attempt to validate app usability and feasibility?
7. Does the app have a clinical/recovery foundation relevant to your intended use?

To recap, if an app has satisfied criteria in Steps One and Two within this Model, then you may assume that:

1. It offers minimal risk in terms of digital safety and privacy.
2. It appears to have some benefit.

Thus, Step 4 helps you to evaluate usability, because an app is only as useful as determined by you and your patients, after taking it for a spin. Because of this, Usability is a more subjective category and so different people will have very different ideas about what this means to them. The questions below are, again, designed to help you think about the app's interface and overall functionality and then make an informed decision about how usable an app will be for the case and patient at hand.

1. What are the main engagement styles of the app?
2. Do the app and its features align with your needs and priorities?
3. Is it customizable?
4. Does the app clearly define functional scope?
5. Does the app seem easy to use?

Finally, the last step in the model is Data Integration towards Therapeutic Goal. This is the topmost level, as the ability to share data may only matter if this is an app that you and the patient want to use (based on background information in Step 1); if it is safe and secure (Step 2); has some evidence base (Step 3), and is easy to use (Step 4). The reason why data integration with the patient's therapeutic goal becomes important in this model, is because apps should not fragment care and the patient and psychiatrist should be able to share and discuss data, or retrieve feedback from the app as appropriate.

In some cases, the ability for apps to share data may not be relevant. For other apps, however (e.g., mood trackers and medication management), ensuring that such data can be easily shared and accessed by those who need to see it is an important factor to consider. While the specifics of this level in the Model will vary for each patient (e.g., with respect to the devices they use, and your medical record system) the following questions can help you to think about whether an app's data or other output can be used in a clinically meaningful way.

1. Do you own your data?
2. Can data be easily shared and interpreted in a way that's consistent with the stated purpose of the app?
3. Can the app share data with EMR and other data tools (apple Healthkit, FitBit)?
4. Is the app for individual use or to be used in collaboration with a provider?
5. If intended to be used with a provider, does the app have the ability to export or transfer data?
6. Does the app lead to any positive behavior change or skill acquisition?
7. Does the app improve therapeutic alliance between patient and provider?