UoP Logo

Digital Education team blog

,

QMP Recently Asked Questions

Sid White (Learning Technologist), Joan Gavin (Learning Technologist), Carolyn Gentle (Learning Development) at Plymouth University and Vivienne Shaughnessy at Questionmark answer questions raised in a recent UP QMP user survey:

Contents:

1: Setting up a test

2: Test appearance

3: Test functionality

4: Interoperability

Appendix: 1h: Personalising feedback

1: Setting up a test

1a: I am interested in starting out with QMP, is it difficult to use?

Carolyn: Academics who want to assess their students using QMP are offered a one-to-one introductory session with a specialist learning technologist who will make sure that all necessary permissions and access are in place as well as giving a step by step guide to what you can do and how to do it using the technology. This session aims to overcome the sticking points that some users experience when using the technology but also provides help in the tricky business of practising a different approach to assessment. Everyone is aware of the high stakes involved in setting and moderating summative assessment which means that there is less scope for a trial-and-error approach, this may contribute to a heightened awareness of difficulty and risk.

1b: I was shown how to set up a question bank but have now forgotten, is there a quick tutorial I could access to remind me?

Joan: the online tutorials in the UP Perception Users Community (found on the staff portal) include guidance on this. See: https://staff.plymouth.ac.uk//percuser/P4OnlineT/intranet.htm , Step 4 addresses this issue.

Sid: We are in the process of producing a set of podcast and vodcast mini-tutorials to support you. They focus on use of the new version of QMP: version 5 which will be introduced at UP early in 2012. By using the podcasts, you will be both reminded of what you learned earlier and upskilled in the use of QMP v.5.

1c : Creating questions is time-consuming, is it possible to access and use prepared questions?

Sid:  Some faculties share questions amongst themselves. Whilst you are best-placed to instigate collaboration with your colleagues on question development, it is also possible for us in Technology Enhanced Learning to give you access to suitable question libraries.

Another solution is the one adopted by PMS: the Medical School pays students to create questions which are then submitted to rigorous quality checks before being added to question banks for future use. They now have a very large bank of library questions and an efficient system for maintaining an up-to-date supply of relevant, high-quality questions.

Please contact the Learning Technology team (x87600) if you would like to discuss gaining access to a multiple user library folder.

1d : Is it difficult to set up folders?

Sid: it can be difficult to do this. Unfortunately this is the way the system works, but I can assure you, we will do everything possible to make it as painless and trouble free for you as possible. Just get in touch if you are having problems setting up folders.

1e : Can I delete questions or assessments easily?

Sid: Users can delete anything and everything on their Local Repository. The difficulty is that those who have rights on the Shared Repository (the live server) automatically get permission to delete their root folder which could lead to real problems – not only for that individual but for other Users as well. The decision was taken early on to prevent this by restricting the rights to read, edit etc. I would really like to give the Users delete permissions, because I know it’s a nuisance. Joan and I are going to make a point of looking into with v5 and try and find a way around it. When we have done that, we will contact all Users.

1f : I would like to use fill-in the blanks questions, is this possible?

Joan: QMP can do this, but if you want QMP to mark the answer, it can be problematic because you have to think of all the different answer parameters – not only different words but also variations such as spelling errors- when setting up the question in the first place so that all acceptable answers can be recognised by the software.

You also have the option that free text answers can be marked by hand in the same way as an essay question.  This doesn’t mess up the marking although those students who see a final mark at the end of the test would need to be advised that some element(s) of the mark were yet to be added. A message at the end of the test could remind them of this.

1g : I would like to develop a set of more advanced questions, is this going to be problematic?

Sid: Without knowing just what kind of questions you have in mind, it is difficult to give a detailed answer to this question but our mini-tutorial podcasts (currently in production) will be available to you and they will offer suggestions for question-types which you might be keen to try using. Don’t forget that the LTs can be contacted with this kind of query and can offer just-in-time advice and guidance if you encounter a difficulty whilst developing a set of questions.

1h : I would like to personalise feedback, how do I do this?

Carolyn: It is really a question of having the steps to hand and then following them in order to achieve what you are aiming for. Since there are server variables that can be used for questions, assessments or assessment outcomes, the detailed guidance from QMP for what the options are and how to use Author Manager to achieve what you want to, is included as an Appendix at the end of this document.

2: Test Appearance

2a : I find QMP is very clunky with graphics and mathematical expressions, will QMP version 5 be better than QMP version 4?

Sid: In the past, there has been an issue with some graphics not going across to the server, we found that this was due to the wrong type of extension or illegal symbols in the graphic name. What would be an improvement of course would be for a warning message to be displayed if and when this happened. We are currently checking QMP v.5 to see whether this continues to be an issue.

Vivienne: Math formulas are in the development pipeline, [they] will soon be available in Questionmark Live

‘Perception provides a sophisticated tool in Authoring Manager called the MathML Editor. This tool enables authors to include mathematical equations of any complexity in their questions. Mathematical equation can be built up from palettes that provide simple symbols and templates for more complex terms.

An equation is stored in the repository as both a JPEG image and in MathML format. When an equation is displayed, the JPEG image is used, but when it is edited the MathML code itself is updated.’ http://www.questionmark.com/perception/help/v5/product_guides/authoring/default.htm Mathematical Equations section.

2b : Web layout can be problematic, if I developed my use of QMP, would I find that I could customise templates and layout quite easily?

Sid: All UP Perception Users have template editing rights which allow them to do this. ‘Assessment templates determine what the assessment will look like. They are used to control the font type, font size, layout and functions within an assessment. Perception comes with a number of default templates. In addition to the default templates, additional templates can be created and edited using Authoring Manager.


… Most organization (sic) update the existing templates to be inline (sic) with their corporate colors and often include logos and other images identifying the company. One template is usually used for a specific type of assessment delivered by an organization, so it is important that the authors understand the affects any modifications can have to the assessments that are delivered to the participants, especially when there is only a single Perception Server in use. Because of this it is recommended that authors are not given the access rights required to edit templates and that existing templates currently in use are not modified but instead new ones are created so that changes and modifications can be tested before deployment. This is particularly important if using multiple repositories because any changes created in a local repository to a template will be copied to the other repositories when a Qpack is imported.’

http://www.questionmark.com/perception/help/v5/best_practice_guides/multi-author/content/using_assessment_templates.htm

Further note about templates’ transition to QMP v.5

‘If you have upgraded to Perception version 5 from Perception version 4, you may need to upgrade the version 4 templates you have been using before they will work correctly with Perception version 5. You will need to upgrade your version 4 templates if:

  • You have modified your Perception version 4 templates by editing the template files directly or by using the Template Editor in Authoring Manager
  • You have created new version 4 templates

You will NOT need to upgrade your version 4 templates if you have not made any modifications to the version 4 templates. If this is the case, when you upgrade to Perception version 5, the version 4 templates being used will be automatically upgraded to version 5 templates, and you will not need to follow the instructions given in this chapter.  It is recommended, however, that you check that your assessments still work the way you expect.’ http://www.questionmark.com/perception/help/v5/product_guides/templates/Upgrading_templates/Overview.htm

3: Test functionality

3a : In terms of risk in e-assessment, I am particularly concerned that students don’t lose any of their work whilst they are doing a test. I have heard that this sometimes happens with QMP, can you reassure me on this point?

Vivienne: Improved Reliability – In the new version [QMP v5] any attempt to leave an assessment once it has been started will display a warning message

Carolyn: One of the greatest problems for students taking Computer Aided Assessments (CAA) is the human hazard of plugs being knocked out of sockets etc. Specially designated CAA areas aim to keep this risk small for example by locating electrical sockets and wiring away from feet. Any pre-test advice to students which helps to minimise the risk that they present to themselves, both physically and in terms of them having a clear understanding of what they can and can’t do whilst taking the test (closing the browser, clicking the back button, saving their work etc) is bound to reduce the risk by raising awareness.

3b : How do I adjust or normalise results for new assessments?

Sid: I would suggest checking out ‘Help > Help Contents’ then looking for the information on the Score Editor (bottom of help page).


 

3c : I do not want to publish student results on the portal. Is there any way I can control this?

Sid: This will be resolved in QMP v.5 as we will have more servers which can be dedicated to different usages. Currently we have just one server distributing assessments and reports, so our ability to vary delivery is limited. More servers will allow us to change the functionality of the server to meet the needs of different users.

3d: Is it possible to deactivate the back button to prevent accidental logging out mid-test?

Sid: The back button has been a problem up to now as students can lose work and log out by clicking on it. In QMP v. 5 a warning appears on the screen when students attempt to go back so they have to choose to override the warning in order to actually log out of the test.

3e: I am worried about students getting timed-out, how can I avoid this?

Sid: You have the option to add time restrictions when you create the assessment and then again when you create the schedule. You can if you want, restrict the amount of time the assessment is presented to the student or leave it so they have constant access, it’s really up to you.

3f : The university network can be very slow, does this compromise the test process?

Sid: This is a combination of a number of things including server setup and usage but there are many more factors. I can tell you that QMP v.5 will be setup differently and will be a lot more powerful than the current specification.

Carolyn: UP QMP users have overcome this potential problem for summative assessments in two different ways: firstly they have the computers ready to go when the students arrive so that students don’t need to login to their own area but login directly to the test; secondly, the test timing does not start until the student clicks on ‘start’ at which point the individual timing of the test is activated.

4: Interoperability

4a : When there is a change from QMP version 4 to QMP version 5, will I have a lot of unlearning and re-learning to do?

Sid: No, the versions have many of the same functions but version 5 should have some nice additional features such as a language translation tool – not available in version 4.

4b: Is it possible to link directly to a scheduled test from the student portal and integrate with calendar and email in order to send students a reminder?

Sid: This isn’t available in QMP in its current server setup. But using the ‘Share Point Connector’ on the new version 5 servers could link QMP with ‘Outlook’. At present permissions have not been granted to send e-mails from the QMP server. We have requested that e-mailing [Simple Mail Transfer Protocol (SMTP)] be made available in QMP v5. This would allow e-mails from the server to be sent to the students giving their results along with feedback.

4c : Is QMP compatible with popular browsers, laptop systems and word processors when creating questions?

Sid: The most popular browsers are all compatible with QMP. That includes Internet Explorer 7 and 8, Firefox; Safari and Chrome. Please note that QMP is not compatible with Internet Explorer 6 or Safari on Windows machines.

4d: How can I ensure that students taking summative tests are using a suitable browser?

Joan: If students are taking their tests on networked machines on campus then their access to a suitable browser should not be an issue. The following are all compatible: Explorer 7 and 8, Firefox; Safari and Chrome. Clearly the easiest way to avoid this being an issue during the test is to highlight to students beforehand exactly what you want them to do – including which browser you want them to use. Alternatively, the invigilator could do a quick visual check before the test begins.

4e : How difficult is the server/client arrangement to use? Do I have direct access to questions on the server?

Sid: It is not too difficult – you just need to create an asset and then cut and paste it onto the server. Yes, you do have access to the questions on the server, you can go in and edit them, the only thing you can’t do is delete them – this is a protective measure designed to avoid material being deleted in error.

4f : How do I ensure that there is no duplication of questions in an exam?

All users have direct access to the QMP server, their Topics (question bank) and Assessments. Questions can be duplicated either by creating identical questions with different ID’s or by asking the software to choose a question from a question bank (Topic folder) for two or more assessment blocks of questions.


Take this situation where ‘Block one’ asks the Question bank for a number of random questions. The student is presented with these questions, they complete the block and move onto the next block of questions within the assessment. The assessment then receives more random questions for the student from the same Topic ‘Question bank’ as before.

When the software has to choose random questions, particularly if there aren’t a lot of questions to choose from, the chances are that in the questions selected could again be one or more of those selected earlier.

Foresight is required to prevent duplicate questions showing in assessments and I know this is difficult to predict. If this is something you are concerned about then only use the question bank once, but if you have to, don’t leave it up to the software to choose the questions. Alternatively make a subfolder where you can keep generic questions within the main folder.

4g: Can I be confident that student group lists are accurate?

Sid: The accuracy of the student group list is the responsibility of the faculty office which updates the students’ records in UNITe. These are then passed onto the Active Directory and then linked to QMP.

4h : Can I control the number of browser windows a student has open during a test?

Sid: When the QMPv5 system is setup in early 2012, we will be instigating QMP Secure during the setup.

‘Question mark Secure helps prevent cheating on high-stakes exams delivered online by disabling functions that participants could use to print or copy exam material, ‘accidentally’ exiting a test, or gaining access to materials on their computers or the Internet that could give an unfair advantage. When a participant takes an online assessment using Questionmark Secure, the secure browser displays the HTML content of the assessment but disables key functions such as task-switching, right click options, screen captures, menus and printing.’ http://www.questionmark.com/us/perception/questionmarksecure.aspx

4i: Time and paper are wasted when paper copies are produced for every test in case of a system crash, is there any way to avoid this?

Sid: Yes, the majority of us are aware of the need to reduce the amount of paper wasted within the University. The UP Computer Assisted Assessment Guidelines for Staff state that:

‘ Paper copies of a version of the examination should be available in case of problems with individual students during the examination. There should be copies available for 5% of the total group, with a minimum total number of 10 copies. These should be on coloured paper and be returned to the invigilators at the end of the session’

( https://staff.plymouth.ac.uk//percuser/summassguide/intranet.htm Before the examination, paragraph 3).

Whilst it may seem frustrating to have to produce these copies at all, this has to be offset by the reassurance they provide and the knowledge that good practice in assessment is being adhered to. In many cases it means producing 10 copies and, in the likely event that they are not used in the assessment, you may choose to recycle them later by offering them to students as a formative test.

Appendix 1: Personalising feedback in QMP

This section contains a list of the server variables that can be used in definition values. These variables can be used in questions and assessments to create customized messages to the participant taking the assessment. This Knowledge Base is split in to the following sections:

Also available in this Knowledge Base is information about how to include server variables when using Authoring Manager. This can be found in the following section:

Server variables that can be used in questions

Variables as it appears in question Name in Authoring Manager Description
%SESSION.TITLE% Assessment name The name of the assessment as defined in the shared repository
%SESSION.ID% Assessment ID The unique numeric ID identifying this assessment in the shared repository
%SESSION.TIME% Time The current time
%SESSION.DATE% Date The current date
%SESSION.USER% Participant name The current participant name, or blank if not set
%SESSION.USERDETAILS% Participant details This returns the content of the participant details field from the shared repository. If this field is empty or if the participant entered via open.php a blank string will be returned
%SESSION.GROUP% Participant group The group of the current participant, or blank if not set
%SPECIAL.1% – %SPECIAL.10% Special 1 – Special 10 Contains any demographic data that is assigned to Special Field 1 through 10

Server variables that can be used in assessments

Variables as it appears in question Name in Authoring Manager Description
%SESSION.TITLE% Assessment name The name of the assessment as defined in the shared repository
%SESSION.ID% Assessment ID The unique numeric ID identifying this assessment in the shared repository
%SESSION.TIME% Time The current time
%SESSION.DATE% Date The current date
%SESSION.USER% Participant name The current participant name, or blank if not set
%SESSION.USERDETAILS% Participant details This returns the content of the participant details field from the shared repository. If this field is empty or if the participant entered via open.php a blank string will be returned
%SESSION.GROUP% Participant group The group of the current participant, or blank if not set
%SESSION.TIMELIMIT% Time limit The time limit of the assessment in seconds, or 0 if no limit
%SPECIAL.1% – %SPECIAL.10% Special 1 – Special 10 Contains any demographic data that is assigned to Special Field 1 through 10

Server variables that can be used in assessment outcomes

Variables as it appears in question Name in Authoring Manager Description
%SESSION.TITLE% Assessment name The name of the assessment as defined in the shared repository
%SESSION.ID% Assessment ID The unique numeric ID identifying this assessment in the shared repository
%SESSION.TIME% Time The current time
%SESSION.DATE% Date The current date
%SESSION.USER% Participant name The current participant name, or blank if not set
%SESSION.USERDETAILS% Participant details This returns the content of the participant details field from the shared repository. If this field is empty or if the participant entered via open.php a blank string will be returned
%SESSION.GROUP% Participant group The group of the current participant, or blank if not set
%SESSION.SCORE% Score in points The score for the assessment so far
%SESSION.MAX% Max score The maximum score for the assessment so far
%SESSION.TIMELIMIT% Time limit The time limit of the assessment in seconds, or 0 if no limit
%SESSION.TIMEELAPSED% Time so far This is the number of seconds elapsed since the assessment started, as the server measures it
%SESSION.TIMELEFT% Time left This is the number of seconds left until the time limit is reached, as the server measures it. If there is no time limit, it is –1
%SPECIAL.1% – %SPECIAL.10% Special 1 – Special 10 Contains any demographic data that is assigned to Special Field 1 through 10

Adding variables using Authoring Manager

Variables can be added in the following locations:

  • In questions
  • In the assessment content
  • In the assessment outcome

How you add the variables to these locations is the same, the only difference is where you add them. In the following example we will add server variables to the outcome of an assessment that explains to a participant how long the took to answer the assessment.

  • In Authoring Manager create a new assessment and add the necessary questions to it
  • Right click on the assessment you just created and click Edit Assessment…
  • Select the Feedback in the Assessment Outcomes section
  • Click Edit Selected Item from the Context menu
  • Make sure the Outcome tab is selected in the Settings menu

  • Click the Expand button to show the rest of the Formatting Toolbar
  • Select the variable from the drop-down list
  • Once you have finished adding the necessary variables and text click OK followed by Save And Exit

Now when you view the feedback for the assessment you should have something similar to the following displayed:

,

Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.