Client Data Sharing:
Lessons Learned from Marin County's CIAP Project
| Introduction |
The past ten years have brought enormous changes to the traditional system of social services. Service providers have had to develop new strategies to meet this new environment. One key strategy has been collaboration—an unprecedented cross-agency (and sometimes even cross-sector) coordination of effort. Another key strategy has been a focus on client information management for increased accountability and analysis of clients' needs and service benefits. These two ideas have presented great opportunities, as well as great challenges, to agencies struggling to serve the public more effectively.
This report is about the combination of these two trends. How can social service collaborations collect and analyze data on their shared clients and services? Such "client data sharing" projects are more and more common, yet there do not appear to be any established best practices for this new and challenging field. With this report, we hope to present our own lessons and experience for the benefit of others. Although the authors of this report have worked with such projects since the early '90s, the focus and example here will be a single data collaboration project in Marin County, California during 1998-1999.
Marin is a suburban/rural county in the San Francisco Bay Area, home to some 240,000 residents. With high rents, small rental housing stock, and close proximity to urban population concentrations, the county has a significant rate of homelessness, yet the problem is largely hidden by the area's overall affluence.
In 1993, twenty providers of housing and related services in Marin County came together to form the Marin Continuum of Housing and Services (MCHS). MCHS was created with the support of the Marin Community Foundation (MCF), one of the nation's largest community foundations, and a funder of most of the MCHS agencies. The mission of MCHS was to create a seamless, comprehensive continuum of services to assist homeless and low-income residents to gain financial and social self-reliance.
Early on, MCHS realized that this mission required more information about the group's clients and services. For example, there was no accurate, unduplicated count of homeless people served by the participating agencies. In response, the group set up a project (the Community Inter-Action Partnership, or CIAP) to develop a system for compiling and analyzing client data from MCHS's member agencies. CIAP did some excellent preliminary work, including the development of a shared intake form and critical "trust-building" between agencies. But—because of staffing shortfalls and lack of clear protocols, among other reasons—the early tools and plans were never fully implemented.
In 1998, MCHS retained a team of consultants to help move CIAP forward. Josh Senyak of Quicksilver Consulting and Sally Brown of Philliber Research Associates worked with MCHS in 1998 and 1999 to develop and implement a system for compiling and analyzing basic client data. The bulk of this report describes the system's creation over this period, its development, and key lessons and best practices gleaned from the project. The results of the data collection and analysis in 1999 are attached as the report A Clear and Present Crisis.
| The Client Data Sharing Process: A Model |
The following model was designed by Josh Senyak, a consultant who works with client data sharing projects and teaches data sharing theory and techniques. It shows the main steps and challenges likely to arise in client data sharing projects. Typically, groups involved in these projects must work through seven key steps:
| 1. Identify your goals. Each goal must be something that makes a concrete difference in clients' lives or in people's understanding of a particular issue. (Thus, "having a client database" is not considered a goal.) Usually it takes the form of a specific question to be answered or a specific change in the way services are offered. Examples include an unduplicated client count; a client demographic profile; a shared intake process; even shared case management between agencies. 2. Identify the participants. Explicitly identify the participating organizations and the participating clients. Which clients should be included in the data collection? Which should not? Do all participating agencies identify these clients in the same way? (For example, if the project addresses homeless clients, do agencies have a single definition of homelessness?) Which agencies must participate in the project for it to reach its ultimate goals? 3. Identify the information to be shared. Your challenge is to define—in detail—the minimum data set needed to achieve your goals. Good information is expensive. Every piece of data incurs a cost—for data collection, data entry, system maintenance, reporting, system design, and staff training. The less information people must process, the better the chances of its being accurate and complete. Be relentless on the distinction between what people want to know from the system and what they need to know. 4. Design the implementation Once the goals, participants, and data are identified, it's time to plan the details of how the data will be collected and managed. Your group will have to resolve key issues, including client identifiers, confidentiality and security, system change, roles of participants, and reports needed from the system. This is also the time to develop or purchase tools (including paper forms, database software, and training materials) and to write up protocols. 5. Test the process No matter how well you plan, some aspects of the project just can't be predicted until you try it out in the field and shake out the bugs. A test period can also be an opportunity for providers to grow comfortable with the project, perhaps in a "commitment-free" pilot of a few months. 6. Share information The payoff. Solicit feedback from the pilot test, incorporate it into your process, get renewed commitment from the participants. Then start collecting, managing and reporting on shared client data. 7. Evaluate and expand Review the project: How did it go? What worked and what didn't? How closely did you meet your goals? This is also the time to consider expanding your goals, participants, clients, or data collected... effectively spiraling back up to the beginning of the process. |
Of course, this seven-step process is only an idealized model. Real-life data sharing projects don't always stay between the lines. The CIAP project (like most real projects) eventually completed every step—but only after it started somewhere in the middle, skipped steps, and eventually returned all the way to the beginning. The next section, "The Lessons," presents some of the specific, practical lessons that participants identified when looking back at the CIAP project. We've grouped the lessons by theme, in more or less chronological order. This lets us correlate steps in the process with the relevant set of lessons. Here is the correlation, for readers who like to follow that level of structural detail:
| Step of the Process | Relevant Lessons |
|
1. Identify your goals 2. Identify the participants 3. Identify the information to be shared |
1. Shared Goals and Vision 2. Project Scope 3. Group Process 4. Using Consultants |
| 4. Design the implementation |
5. Major Challenges and Solutions 6. Technology |
| 5. Test the process | |
| 6. Share information | |
| 7. Evaluate and expand |
7. Project Results and Value 8. What's Next? |
| The Lessons |
1. Shared Goals and Vision
The CIAP project grew out of the vision of the Marin Community Foundation (MCF), a key funder of human services in Marin County. MCF, eager to see housing providers working together more efficiently, supplied both "stick" and "carrot": a mandate for the agencies to work together, as well as funds specifically for inter-agency collaboration. This led to the establishment of the MCHS collaborative in 1993.MCHS was formed to work on overall service coordination. A "client data system" was seen only as a means to this goal, not an end in itself. This distinction is critical. The group worked on plans for data sharing for several years, but before becoming a reality the idea had to be framed in terms of very clear, concrete benefits for the service providers and their clients.
As it turned out, the group's most immediate data need was a single, unduplicated count of people—homeless or at risk of homelessness—served by participating agencies. Policy-makers and planners had long sought an accurate and unduplicated count of homeless people in the community, but the decentralized nature of Marin's service delivery system made it very hard to arrive at such a number. There was a semi-official estimate of 3,000 homeless people in Marin, but in fact nobody knew whether this number was accurate, nor were any demographics available for the population. The lack of hard data hampered planning efforts, and had led in at least once case to loss of federal funding opportunities. The CIAP project moved into high gear as soon as MCHS set this as its first goal—an unduplicated cross-agency count of clients, with a basic set of demographics on those clients.
This goal was successfully met by Phase I of the CIAP project. The group was able to produce hard data showing that the county's homeless population was much higher than previously estimated. They were also able, for the first time, to analyze patterns of their clients' housing status, income, ethnicity, age, household size, and special needs.
In Phase II, MCHS began to define new, expanded goals for the data system, involving turnaways and outcomes. "Turnaways" are client requests for services that can't be met because of insufficient resources or capacity. "Outcomes" reflect the eventual changes in the lives of those who receive services. These expanded goals are expected to be implemented in Phase III of the CIAP project, in the years 2000 and 2001.
Lessons:- Refine the vision. The funder's original vision was relatively broad and unfocused in its scope. Although this breadth is appropriate and necessary, it took several years for the direct service providers to refine and sharpen the vision, making it realistic, concrete, and specific to their own needs. The project could not move forward until this refinement and clarification was complete.
- Focus on benefits to clients. Generally, providers agreed to participate in the project when they saw that they and their clients would benefit from its outcomes. Even so, getting full "buy-in" from the providers took considerable time and effort. This work was greatly helped by a clear, simple statement of goals and benefits.
- Build from a foundation of service collaboration. Client data systems are sometimes touted as a means of strengthening service collaboration. In reality, it's just the reverse: the data system can only be as strong as the collaboration it serves. A sophisticated data system demands a very high level of collaboration—limited points of entry; shared forms for intake, service and assessment; formalized referral and follow-up processes; etc. A looser, more diverse collaboration can support only a relatively simple client data collection system.
- Set clear and reasonable goals. The goals for data sharing must be simple and clear. CIAP's first goal was an unduplicated client count and basic client demographics; having achieved that, it is now beginning to add data on turnaways and client outcomes.
2. Project Scope
In 1997, MCHS made its first attempt at data-sharing. A committee of the providers developed a ten-page client intake form for shared use. This form was essentially a combination of all the providers' intake forms into a single instrument. Although some client information was collected, this early attempt did not, finally, yield useful data. Data collection goals and procedures were unclear; the instrument was found to be cumbersome; and agencies differed in their level of commitment to the project.In 1998, after retaining its consultants, the group planned and successfully completed the first phase of the CIAP project. This phase was a three-part process:
-
The group investigated other collaborations' attempts at client data sharing. For example, representatives from a HIV services collaboration in San Francisco and from a human services collaboration in Contra Costa County gave presentations on the successes and pitfalls they had encountered in similar projects.
- The consultants helped the group plan and implement a limited pilot data collection. To this end, they developed a simple one-page client information form, a database, a reporting format, and a data collection protocol. The providers received training and collected six months' worth of client data.
- The group planned the continuation and expansion of the project in 1999.
In 1999 ("Phase II"), the group expanded the system to include data on agencies' existing clients, allowing a full count of the number of homeless and at-risk households served by MCHS during the year. (In Phase I, data had been collected for new intakes only). MCHS also began outreach to relevant non-MCHS agencies, encouraging them to join the data collection process for a more complete picture of needs and services in the county. Phase II also reflected MCHS's commitment to compiling and disseminating the lessons it has learned so far, for the benefit of other groups around the country planning similar projects. Finally, the group examined the possibility of collecting data on services, turnaways, and outcomes, and is now (mid-2000) implementing some of these plans in Phase III.
Lessons:- Expect mistakes. False starts are likely and perhaps even necessary. Having tried unsuccessfully to collect data in 1997, the MCHS providers were more sophisticated on their second try. In particular, they were convinced of the need to streamline their goals and simplify their system.
- Proceed in small steps. Don't try to accomplish a "dream system" all at once. Break the work down into manageable pieces. Each step helps to develop the collaboration, build trust among the providers, and "reality-test" the benefits and performance of the system.
- Clarify your options. "Client data sharing" can mean nearly anything—from informal phone and fax referrals, to a simple system that aggregates statistics from paper forms, all the way to a real-time desktop system that maintains client and service information to support cross-site case management. Collaboration members must be very clear about the differences between the various options, to avoid confusion or unrealistic expectations.
- Learn from others. Reports or presentations from other groups that have tried data sharing can be very helpful in making the project more concrete to collaboration members.
3. Group Process
Participants noted two especially helpful aspects of the process in Phases I and II. First, the process was highly inclusive; participants reported that their concerns, needs, and ideas were taken seriously. The consultants conducted individual interviews with all of the participating agencies, to gain a better understanding of their services, their client flow, their paperwork, and their individual concerns. The goals, tools and methodologies were presented as works-in-progress, with multiple opportunities for input from all participants.Second, the process was iterative. The data collection system was reviewed and revised after the pilot, and then again in the course of Phase II. These revisions covered the database tools as well as the process, procedures, and reports. The project included a strong feedback loop, so that the system could be continually readjusted to meet the overall project goals.
Lessons:- Listen. Be sure that all participants' points of view are heard and taken seriously.
- Improve continually. Treat the project as a work-in-progress rather than as a fixed and final product. Plan to put time and resources into ongoing improvement of every part of the project.
- Set a reasonable pace. The pace of decision-making is an important balancing act. On the one hand, the pace must be slow enough that the group can move forward as a whole. Because this is a new area for many participants, people will need time to think through the practical details and their consequences. On the other hand, the pace must be quick enough to keep momentum and show real successes. The project may have to move forward with the largest committed group, rather than waiting for everyone to sign on. Early results and successes can help bring in the others, later on.
- Seek "top-to-bottom" agency participation. Try to get buy-in from every level of the participating organizations, from the executive directors to the front-line staff. These different stakeholders will have different levels of involvement, and different reasons for being involved with the project, but it's important to get the full range of perspective and commitment.
4. Using Consultants
CIAP participants reported a positive experience with their project consultants. Selecting experienced consultants was a first step in kicking the project into high gear, leading to concrete successes in Phases I and II. Participants noted that, before the consultants had been hired, the project tended to lose momentum between monthly meetings, since none of the participants had primary responsibility for carrying the project forward and following up on tasks.Consultants can bring a very useful, specific set of skills and expertise to a data collaboration project. Unlike other participants, they have primary responsibility for keeping the project moving forward and meeting goals and deadlines. As "outsiders," they may bring an objectivity that can help guide the group through inter-agency politics.
Quicksilver and PRA, MCHS's consultant team, brought expertise in evaluation, technology, and nonprofit organizational development, as well as extensive experience with social service delivery and data-sharing collaborations. This broad range of experience proved invaluable. A consultant specializing in only one or two of these areas might not have met this project's complex needs.
Lessons:- Be selective. In selecting a consultant, be sure to check recommendations and references very carefully. Look for concrete demonstration of the skills and experience that the consultant claims. Some of the most important characteristics of a consultant for a data collaboration project include:
- Experience with cross-agency data collection; working familiarity with different models, best practices, approaches, issues, and solutions.
- Expertise in technology, together with a strong awareness of the limitations of computer technology.
- Some background in the providers' programmatic field (e.g. homelessness, child care, medical).
- Excellent skills in facilitation, organization, writing, and making numbers meaningful for project goals.
- Experience with cross-agency data collection; working familiarity with different models, best practices, approaches, issues, and solutions.
- Keep decision-making authority. The consultant should not be responsible for making project decisions, but rather for laying out the options, consequences, costs and benefits to help your collaboration define its own goals and make its own decisions.
- Set a realistic scope. Consultants are most effective when they are given a small number of clear, realistic deliverables. Too many deliverables, or deliverables which are vague or unrealistically ambitious, suggest that more planning and prioritizing is needed before the consultant is engaged.
- Know the agencies. Consultants should meet individually with representatives from each agency, in order to understand the full range of paperwork, processes, and level of buy-in to the collaboration project. It's hard to help providers without understanding their particular challenges; structure, staffing, intake process, client needs, technology readiness, etc. (This can be especially difficult for consultants who lack experience in the nonprofit sector.) An individualized approach also helps allay providers' fears of an inappropriate "one-size-fits-all" approach.
- Don't leave everything to the consultant. Several important roles must be filled from within the collaboration, rather than by consultants. For example, someone within the group must take leadership as liaison with the consultant—providing a clear point of contact, as well as identifying and resolving complex inter-agency issues that may come up. There's also a critical "quality control" function (data checking and cleaning, collecting and compiling the data, training new staff to use the system, etc.) that is probably best assigned to one of the collaborating agencies, in the interests of long-term sustainability.
5. Major Challenges and Solutions
Two major issues came up early and often in CIAP planning. The first issue was confidentiality. Providers had a number of concerns. How could we get unduplicated client counts across agencies without breaking confidentiality as to client identities? How would computerized data be protected from unauthorized view? If client data was to be stored on computers equipped with modems, would this make it somehow vulnerable to hackers via the Internet?Our approaches to the confidentiality issue were both technical and procedural. One of the technical solutions was the definition of an identification code, made up of the first two letters of the client's last name plus the client's full birth date. This code is nearly unique for the size of the population involved; that is, it would be reasonably unlikely for two different homeless people in Marin County to have the same code. At the same time, it is reasonably difficult to extrapolate from the code to a particular individual. The databases were password protected, and shared paper forms kept in locked cabinets.
Procedurally, the group adopted a pragmatic philosophy: confidentiality of client data did not need to be "bullet-proof"; it simply needed to be as good as or better than agencies' existing security measures for client information. The group examined the project's client data processes step by step, and confirmed that at every step the computerized data was safer than it was in its conventional form. In other words, if anyone wanted to know whether client X was receiving services at agency Y, it would be easier to learn this by physically breaking into the agency and stealing the paper files than by intercepting collaboration data. A key to this approach was helping the participants gain a clear and adequate understanding of the technology involved, in order to demystify common fears of hackers and the Internet.
The other important issue was the data collection burden. Some providers were initially reluctant to add yet another form to the "paper chase" that clients and staff already endured. Communication with the providers was critical in addressing this issue. The consultants met with the providers to estimate the number of forms to be collected at each agency. Some agencies were reassured to see that relatively few forms would have to be completed. Others, facing much larger estimates, found the numbers helpful in planning new staff deployment. The consultants were also careful to keep the data collection form extremely simple, basing it closely on common elements of providers' existing forms. In many cases, the new form could be filled out by clerical staff simply by copying information directly from the agency's existing intake form or face sheet.
Lessons:- Don't minimize the challenges of the project. Encourage participants to speak frankly about their concerns from the outset, and be sure that everyone feels that their issues have been heard and acknowledged.
- Address confidentiality. Three approaches for dealing with confidentiality issues:
- Demystify the technology. People have many fears and misconceptions about technology, which must be corrected with accurate, detailed information. There are many different approaches to sharing client data, each with its own confidentiality consequences. Be sure that participants understand the key differences between the options being discussed.
- Distinguish the concerns. When projects get stuck in arguments over technical details of confidentiality, it may be a symptom of comfort, trust, and ownership issues between the agencies. Technological solutions, however ingenious, can't address these fundamental questions. Get the real issues on the table.
- Delimit the goals. Perfect security isn't possible or necessary. Match your confidentiality goals to the real programmatic needs. Be sure that the new system's security is at least as strong as the current system's. Don't get lost in excessive security measures for shared data while conventional vulnerabilities (such as poorly-locked file cabinets, human errors and carelessness) remain open.
- Demystify the technology. People have many fears and misconceptions about technology, which must be corrected with accurate, detailed information. There are many different approaches to sharing client data, each with its own confidentiality consequences. Be sure that participants understand the key differences between the options being discussed.
- Address the data collection burden. Four approaches for dealing with data collection and paperwork issues:
- Be clear about the costs and benefits. Have a clear goal for the data sharing project that makes it worth the effort. Be prepared to remind participants of that goal as needed.
- Less is better. Keep the forms and process brief and simple.
- Less new information is better. Wherever possible, use participants' existing data collection routines as a basis for the shared data set. Keep the data elements as close as possible to the data that is already being collected.
- Be flexible. Agencies often need time to gear up to full compliance to a new data collection process. Be flexible and creative in helping them get up to speed. Some information is often better than none, especially at the beginning of a project.
- Be clear about the costs and benefits. Have a clear goal for the data sharing project that makes it worth the effort. Be prepared to remind participants of that goal as needed.
6. Technology
Technology should come last in the planning process, not first. Client data collaborations sometimes fail when they start off by focusing on technology, without making the crucial joint decisions about what data should be shared, why, and how. Technology is simply a tool for getting a job done. It needs little attention until the job has been clearly defined.Not to say that technology is unimportant. Once selected, the tools must work reliably and well. Buggy software or crashing networks will erode confidence in the data. Worse, they can interfere with client services and drain scarce agency resources. Work only with reliable vendors and be an educated, proactive customer.
"Show and tell" presentations by other service providers are extremely helpful when it comes to technology. Use the experience of other collaborations. Their stories can make the technical options, possibilities and obstacles much more realistic and concrete to your group.
Lessons:- Resist magical thinking. Technology is vastly oversold by vendors both big and small, aided by the fear, hope, and "magical thinking" of the public. Fight the hype. There's nothing magical about technology. Don't be afraid to ask simple questions about technology, and don't make decisions based on vague or incomplete answers. Talk to others who have gone down the same road before you.
- Focus on goals, not technology. Don't start by assuming that there will be a piece of software at the end of the process. Focus on the project's core goals, not on the technology. What information must the group compile? If a simple paper-and-pencil system can collect that information, use it!
- Be realistic about agencies' capacity. Ideally, your technology solution should match the current technological readiness of the participants fairly closely. If, on the other hand, you expect to move agencies to a new level of technology, be prepared to spend much more time and money on equipment, technical support, staff training, project management, etc. Even so, there's no guarantee that all agencies will make the transition successfully.
- Be a demanding software customer. If you are considering developing or purchasing software, find out exactly how much of the system can be modified; how easy it is to make modifications (and by whom); and how much the modifications will cost. Be sure that software comes with adequate technical documentation.
7. Project Results and Value
After completing Phase I (and most of Phase II) of the CIAP project, MCHS participants took time to reflect on the project's value to date. They defined benefits of the project for education, policy and fundraising; for the participating agencies; and for the collaboration as a whole.
Lessons:- Get value for the larger community. Having accurate client data makes a big difference for education, policy, and funding. County planners and policy-makers gain a more accurate picture of the size and scope of the problem of homelessness. For the first time, funders can get detailed, accurate information on the needs that their resources are addressing. The numbers are also an important tool for community education; certain statistics (for example, the number of homeless children) help to put a human face on the problem, making it more real and concrete for the general public.
- Get value for participating agencies. Individual providers reported benefits from CIAP as well. In many cases, the data analysis gave them a better understanding of their own clients and services. The data proved the surprising amount of need in the county, helping to validate the importance of providers' work. Providers also reported that CIAP data reports were helpful for educating and motivating board members, and for individual agency fundraising.
- Get value for the collaboration. The MCHS service collaboration found that the CIAP process had strengthened its mission in several ways. They found that the agencies had gained a better understanding of each other's clients and services. The work put into CIAP led to greater trust and cohesiveness among the agencies. They also found that the higher level of understanding and cooperation helped to facilitate referrals between the agencies.
8. What's Next?
MCHS plans to continue refining and expanding the CIAP project, with Phase III likely to follow in 2000 and 2001.MCHS will continue outreach to other local service agencies. As more agencies join the data project, the data becomes more complete and thus more valuable. Several geographic and demographic groups in the county will probably be underrepresented in CIAP reports unless the larger non-MCHS agencies join the system.
In Phase III, providers will continue to explore possibilities for adding new types of data to the CIAP system—particularly quantified information on turnaways, outcomes, and services throughout the collaboration.
- Turnaways. We expect the CIAP system to begin collecting turnaway data in 2000. This will be focused on turnaways from certain key services (emergency shelter, permanent affordable housing, and financial assistance related to housing, such as rental deposits and rental assistance). In some cases, to ease the burden on providers, the data will be collected at selected sample intervals; annualized figures will be estimated from the samples.
- Outcomes. The group soon realized that a much larger issue of service philosophy had to be resolved before they could add outcome data to their system. The problem (in brief): the participating agencies diverge greatly in their client populations, services offered, and "intensity" of service models. Yet, to define a set of shared outcomes, providers must agree on some impact that all share as a goal. Even an extremely general outcome—for example, "increased client self-sufficiency"—may not work for all; after all, some agencies work with clients who are so severely disabled that self-sufficiency is not necessarily a goal. In Phase III, the group plans to address these differences in service philosophy. This may lead to definition of a few shared outcomes for the entire group or for selected subsets of providers.
- Service data. The group considered the possible benefits of collecting service data; but, in the end, none of these benefits proved compelling, especially given the substantial time and cost required for service data collection. Most agencies already have good counts of the services they deliver individually. It might be interesting to examine the package of services accessed by a client over the collaboration as a whole, and compare these services to client outcomes, in the hope of finding links between services and outcomes. This option may be explored further if client outcomes are eventually added to the CIAP system.
In all, CIAP participants found data collaboration challenging but achievable and, finally, very much worthwhile. The difficult issues had much less to do with technology than with planning and program considerations. Marin County still doesn't have a sophisticated computerized system that tracks full details on every homeless client that walks into a provider's office. Perhaps it never will. But by focusing on genuine priority goals and breaking the work into reasonable chunks, the MCHS group has managed to implement a simple, functional, inexpensive system that compiles critical information with a minimum of impact on clients and staff.