Dubaiblog925

Vladislav Bolkhovitin

1. Scsi Raid Array
2. Scsi Disk Device
3. Adaptec Scsi Drivers
4. Microsoft Scsi Driver

It worked on Windows 7, and still works on OSX. From the research I did the reason is that up until Windows 7 Intel made the UAS drivers. In 10 they are made by MS and don't work quite the same.so some drives just don't work - it doesn't seem to be one specific manufacturer - see Can anyone use USB attached SCSI in 10? - Windows 10 Forums.

Version 3.0.0 for SCST 3.0.0 and later

1. I needed a U320 SCSI controller after my old motherboard with on-board SCSI had died. The new board lacked such a controller. I searched for hours for a U320 card that didn't have the added cost of RAID.
2. Drivers/scsi/scsi.c¶ Main file for the SCSI midlayer. Int scsichangequeuedepth (struct scsidevice.sdev, int depth) ¶ change a device’s queue depth. Struct scsidevice.sdev SCSI Device in question int depth number of commands allowed to be queued to the driver. Sets the device queue depth and returns the new value.

Appendix

1. Introduction

SCST is a SCSI target mid-level subsystem for Linux. It providesunified consistent interface between SCSI target drivers, backend devicehandlers and Linux kernel as well as simplifies target driversdevelopment as much as possible.

It has the following features:

• Very low overhead and fine-grained locks, which allow to reachmaximum possible performance and scalability that close to theoreticallimit.
• Complete SMP support.
• Performs all required pre- and post- processing of incomingrequests and all necessary error recovery functionality.
• Emulates necessary functionality of SCSI host adapters, becausefrom a remote initiator's point of view SCST acts as a SCSI host withits own devices. Some of the emulated functions are the following:
  • Generation of necessary UNIT ATTENTIONs, their storage anddelivery to all connected remote initiators (sessions).
  • RESERVE/RELEASE functionality, including Persistent Reservations.
  • All types of RESETs and other task management functions.
  • REPORT LUNS command as well as SCSI address spacemanagement in order to have consistent address space on allremote initiators, since local SCSI devices could not know abouteach other to report via REPORT LUNS command. Additionally, SCSTresponds with error on all commands to non-existing devices andprovides access control, so different remote initiators couldsee different set of devices.
  • Other necessary functionality (task attributes, etc.) asspecified in SAM-2, SPC-2, SAM-3, SPC-3 and other SCSI standards.
• Verifies all incoming requests to ensure commands executionreliability and security.
• Device handlers architecture provides extra flexibility byallowing to make additional requests processing, which is completelyindependent from target drivers, for example, data caching or devicedependent exceptional conditions treatment.

2. Terms and Definitions

SCSI initiator device

A SCSI device that originates service and task management requests to beprocessed by a SCSI target device and receives device service and taskmanagement responses from SCSI target devices.

SCSI target device

A SCSI device that receives device service and task management requestsfor processing and sends device service and task management responsesto SCSI initiator devices or drivers.

SCST session

SCST session is the object that describes relationship between a remoteinitiator and SCST via a target driver. All the commands from the remoteinitiator is passed to SCST in the session. For example, for connectionoriented protocols, like iSCSI, SCST session could be mapped to TCPconnection (as well as iSCSI session). SCST session is equivalent ofSCSI I_T nexus object.

Local SCSI initiator

A SCSI initiator that is located on the same host as SCST subsystem.Examples are sg and st drivers.

Remote SCSI initiator

A SCSI initiator that is located on the remote host for SCST subsystemand makes client connections to SCST via SCST target drivers.

SCSI target driver

A Linux hardware or logical driver that acts as a SCSI target for remoteSCSI initiators, i.e. accepts remote connections, passes incoming SCSIrequests to SCST and sends SCSI responses from SCST back to theiroriginators.

Device (backend) handler driver

Also known as 'device type specific driver' or 'dev handler', SCSTdriver, which helps SCST to analyze incoming requests and determineparameters, specific to various types of devices as well as perform someprocessing. See below for more details.

3. SCST Core Architecture

SCST accepts commands and passes them to SCSI mid-level at the sameway as SCSI high-level drivers (sg, sd, st) do. Figure 1 showsinteraction between SCST, its drivers and Linux SCSI subsystem.

4. Target drivers

4.1 struct scst_tgt_template

To work with SCST a target driver must register its template in SCST bycalling scst_register_target_template(). The template lets SCST know thetarget driver's entry points. It is defined as the following:

Where:

• sg_tablesize - allows checking whether scatter/gather can beused or not and, if yes, sets the maximum supported count ofscatter/gather entries
• name - the name of the template. Must be unique to identifythe template. Must be defined.
• unchecked_isa_dma - true, if this target adapter usesunchecked DMA onto an ISA bus.
• use_clustering - true, if this target adapter wants to useclustering (i.e. smaller number of merged segments).
• no_clustering - true, if this target adapter doesn't supportSG-vector clustering
• xmit_response_atomic, rdy_to_xfer_atomic - true, if thecorresponding function supports execution in the atomic (non-sleeping)context.
• no_proc_entry - true, if this template doesn't need the entry in /proc
• max_hw_pending_time - The maximum time in seconds cmd canstay inside the target hardware, i.e. after rdy_to_xfer() andxmit_response(), before on_hw_pending_cmd_timeout() will be called, ifdefined. In the current implementation a cmd will be aborted in time tmax_hw_pending_time <= t < 2*max_hw_pending_time.
• threads_num - number of additional threads to the pool ofdedicated threads. Used if xmit_response() or rdy_to_xfer() is blocking.It is the target driver's duty to ensure that not more, than that numberof threads, are blocked in those functions at any time.
• int (*detect) (struct scst_tgt_template *tgt_template) - thisfunction is intended to detect the target adapters that are present inthe system. Each found adapter should be registered by callingscst_register_target(). The function should return a value >= 0 tosignify the number of detected target adapters. A negative value shouldbe returned whenever there is an error. Must be defined.
• int (*release)(struct scst_tgt *tgt) - this function isintended to free up resources allocated to the device. The functionshould return 0 to indicate successful release or a negative value ifthere are some issues with the release. In the current version of SCSTthe return value is ignored. Must be defined.
• int (*xmit_response)(struct scst_cmd *cmd) - thisfunction is equivalent to the SCSI queuecommand(). The target shouldtransmit the response data and the status in the struct scst_cmd. Seebelow for details. Must be defined.
• int (*rdy_to_xfer)(struct scst_cmd *cmd) - this functioninforms the driver that data buffer corresponding to the said commandhave now been allocated and it is OK to receive data for this command.This function is necessary because a SCSI target does not have anycontrol over the commands it receives. Most lower-level protocols havethe corresponding function which informs the initiator that buffers havebeen allocated e.g., XFER_RDY in Fibre Channel. After the data actuallyreceived, the low-level driver should call scst_rx_data() in orderto continue processing this command. Returns one of theSCST_TGT_RES_* constants, described below. Pay attention to'atomic' attribute of the command, which can be get viascst_cmd_atomic(). It is true if the function called in the atomic(non-sleeping) context. Must be defined.
• void (*on_hw_pending_cmd_timeout) (struct scst_cmd *cmd) - Called if cmd stays inside the target hardware, i.e. after rdy_to_xfer()and xmit_response(), more than max_hw_pending_time time. The targetdriver supposed to cleanup this command and resume cmd's processing.
• void (*on_free_cmd)(struct scst_cmd *cmd) - this functioncalled to notify the driver that the command is about to be freed.Necessary, because for aborted commands xmit_response() could not becalled. Could be used on IRQ context. Must be defined.
• int (*alloc_data_buf) (struct scst_cmd *cmd) - this functionallows target driver to handle data buffer allocations on its own.Target driver doesn't have to always allocate buffer in this function,but if it decided to do it, it must check thatscst_cmd_get_data_buff_alloced() returns 0, otherwise to avoid doublebuffer allocation and memory leaks alloc_data_buf() shall fail. Returns0 in case of success or < 0 (preferrably -ENOMEM) in case of error, or >0 if the regular SCST allocation should be done. In case of returningsuccessfully, scst_cmd->tgt_data_buf_alloced will be set by SCST. It ispossible that both target driver and dev handler request own memoryallocation. If allocation in atomic context, i.e. scst_cmd_atomic() istrue, and < 0 is returned, this function will be recalled in threadcontext. Note that the driver will have to handle itself all relevantdetails such as scatterlist setup, highmem, freeing the allocatedmemory, etc.
• void (*preprocessing_done) (struct scst_cmd *cmd) - thisfunction informs the driver that data buffer corresponding to the saidcommand have now been allocated and other preprocessing tasks have beendone. A target driver could need to do some actions at this stage. Afterthe target driver done the needed actions, it shall callscst_restart_cmd() in order to continue processing this command. In caseof preliminary commands completion, this function will also be calledbefore xmit_response(). Called only for commands queued usingscst_cmd_init_stage1_done() instead of scst_cmd_init_done(). Returnsvoid, the result is expected to be returned using scst_restart_cmd().This command is expected to be NON-BLOCKING. If it is blocking, considerto set threads_num to some none 0 number. Pay attention to 'atomic'attribute of the cmd, which can be get by scst_cmd_atomic(). It is trueif the function called in the atomic (non-sleeping) context.
• int (*pre_exec) (struct scst_cmd *cmd) - this functioninforms the driver that the said command is about to be executed.Returns one of the SCST_PREPROCESS_* constants. This command isexpected to be NON-BLOCKING. If it is blocking, consider to setthreads_num to some none 0 number.
• void (*task_mgmt_affected_cmds_done) (struct scst_mgmt_cmd*mgmt_cmd) - this function informs the driver that all affected by thecorresponding task management function commands have beed completed. Noreturn value expected. This function is expected to be NON-BLOCKING.Called without any locks held from a thread context.
• void (*task_mgmt_fn_done)(struct scst_mgmt_cmd *mgmt_cmd) -this function informs the driver that a received task managementfunction has been completed. Completion status could be get viascst_mgmt_cmd_get_status(). No return value expected. Must bedefined, if the target supports task management functionality.
• int (*report_aen) (struct scst_aen *aen) - this function isused for Asynchronous Event Notifications. Returns one of theSCST_AEN_RES_* constants. After AEN is sent, target driver mustcall scst_aen_done() and, optionally,scst_set_aen_delivery_status(). This function is expected to beNON-BLOCKING, but can sleep. This function must be prepared to handleAENs between calls for the corresponding session ofscst_unregister_session() and unreg_done_fn() callback called or beforescst_unregister_session() returned, if its called in the blocking mode.AENs for such sessions should be ignored. Must be defined, if low-levelprotocol supports AENs.
• int (*read_proc) (struct seq_file *seq, struct scst_tgt*tgt), int (*write_proc) (char *buffer, char **start, off_t offset,int length, int *eof, struct scst_tgt *tgt) - those functions can beused to export the driver's statistics and other infos to the worldoutside the kernel as well as to get some management commands from it.If the driver needs to create additional files in its /procsubdirectory, it can use scst_proc_get_tgt_root() function to getthe root proc_dir_entry.
• int (*get_initiator_port_transport_id) (struct scst_session*sess, uint8_t **transport_id) - this function returns in tr_id thecorresponding to sess initiator port TransporID in the form as it's usedby PR commands, see 'Transport Identifiers' in SPC. Space for theinitiator port TransporID must be allocated via kmalloc(). Callersupposed to kfree() it, when it isn't needed anymore. If sess is NULL,this function must return TransportID PROTOCOL IDENTIFIER of thistransport. Returns 0 on success or negative error code otherwise. Shouldbe defined, because it's required for Persistent Reservations.

Functions xmit_response(), rdy_to_xfer() are expected to benon-blocking, i.e. return immediately and don't wait for actual datatransfer to finish. Blocking in such command could negatively impact onoverall system performance. If blocking is necessary, it is worth toconsider creating dedicated thread(s) in target driver, to which thecommands would be passed and which would perform blocking operationsinstead of SCST. If the function allowed to sleep or not is defined by'atomic' attribute of the cmd that can be get viascst_cmd_atomic(), which is true, if sleeping is not allowed. Inthis case, if the function requires sleeping, it can returnSCST_TGT_RES_NEED_THREAD_CTX in order to be recalled in the threadcontext, where sleeping is allowed.

Functions task_mgmt_fn_done() and report_aen() are recommendedto be non-blocking as well. Blocking there will stop all managementprocessing for all target drivers in the system (there is only onemanagement thread in the system).

Functions xmit_response() and rdy_to_xfer() can return thefollowing error codes:

• SCST_TGT_RES_SUCCESS - success.
• SCST_TGT_RES_QUEUE_FULL - internal device queue is full, retryagain later.
• SCST_TGT_RES_NEED_THREAD_CTX - it is impossible to complete requested task in atomic context. The command should be restarted in thethread context as described above.
• SCST_TGT_RES_FATAL_ERROR - fatal error, i.e. it is unable toperform requested operation. If returned by xmit_response() thecommand will be destroyed, if by rdy_to_xfer(),xmit_response() will be called with HARDWARE ERROR sense data.

More about xmit_response()

As already written above, function xmit_response() should transmitthe response data and the status from the cmd parameter.

Sense data, if any, is contained in the buffer, returned byscst_cmd_get_sense_buffer(), with length, returned byscst_cmd_get_sense_buffer_len(). SCST always works in autosensemode. If a low-level SCSI driver/device doesn't support autosense mode,SCST will issue REQUEST SENSE command, if necessary. Thus, if CHECKCONDITION established, target driver will always see sense in the sensebuffer and isn't required to request the sense manually.

After the response is completely sent, the target should callscst_tgt_cmd_done() function in order to allow SCST to free thecommand.

Function xmit_response() returns one of the SCST_TGT_RES_*constants, described above. Pay attention to 'atomic' attribute of thecmd, which can be get via scst_cmd_atomic(): it is true if thefunction called in the atomic (non-sleeping) context.

To detect aborted commands xmit_response() must in the beginning checkreturn status of function scst_cmd_aborted_on_xmit(). If it's true,xmit_response() must call scst_set_delivery_status(cmd,SCST_CMD_DELIVERY_ABORTED) and terminate further processing by callingscst_tgt_cmd_done(cmd, SCST_CONTEXT_SAME).

4.2 Target driver registration functions

scst_register_target_template()

Function scst_register_target_template() is defined as the following:

Where:

• vtt - pointer to the target driver template

Returns 0 on success or appropriate error code otherwise.

scst_register_target()

Function scst_register_target() is defined as the following:

Where:

• vtt - pointer to the target driver template

Returns target structure based on template vtt or NULL in case of error.

4.3 Target driver unregistration functions

In order to unregister itself target driver should at first callscst_unregister_target() for all its adapters and then callscst_unregister_target_template() for its template.

scst_unregister_target()

Function scst_unregister_target() is defined as the following:

Where:

• tgt - pointer to the target driver structure

scst_unregister_target_template()

Function scst_unregister_target_template() is defined as the following:

Where:

• vtt - pointer to the target driver template

5. Device specific drivers (backend device handlers)

Device specific drivers are add-ons for SCST, which help SCST toanalyze incoming requests and determine parameters, specific to varioustypes of devices as well as actually execute specified SCSI commands.Device handlers are intended for the following:

• To get data transfer length and direction directly from CDB andcurrent device's configuration exactly as an end-target SCSI devicedoes. This serves two purposes:
  • Improves security and reliability by not trusting the datasupplied by remote initiator via SCSI low-level protocol.
  • Some low-level SCSI protocols don't provide data transferlength and direction, so that information can be get onlydirectly from CDB and current device's configuration. Forexample, for tape devices to get data transfer size it might benecessary to know block size setting.
• Execute commands
• To process some exceptional conditions, like ILI on tape devices.
• To initialize incoming commands with some device-specificparameters, like timeout value.
• To allow some additional device-specific commands pre-, post- processing or alternative execution, like copying data from systemcache, and do that completely independently from target drivers.

Device handlers considered to be part of SCST, so they could directlyaccess any fields in SCST's structures as well as use the correspondingfunctions.

Without appropriate device handler SCST hides devices of this type fromremote initiators and returns HARDWARE ERROR sense data to anyrequests to them.

5.1 Structure scst_dev_type

Structure scst_dev_type is defined as the following:

Where:

• name - the name of the device handler. Must be defined andunique.
• type - SCSI type of the supported device. Must be defined.
• parse_atomic, alloc_data_buf_atomic,dev_done_atomic - true, if the corresponding callback supportsexecution in the atomic (non-sleeping) context.
• no_proc - true, if no /proc files should be automaticallycreated by SCST for this dev handler
• exec_sync - should be true, if exec() is synchronous. Thisis a hint to SCST core to optimize commands order management.
• pr_cmds_notifications - should be set if the device wants toreceive notification of Persistent Reservation commands (PR OUT only)Note: The notifications will not be sent if the command failed.
• threads_num - sets number of threads in this handler'sdevices' threads pools. If 0 - no threads will be created, if <0 -creation of the threads pools is prohibited. Also pay attention tothreads_pool_type below.
• threads_pool_type - threads pool type. Valid only ifthreads_num > 0. Possible values:
  • SCST_THREADS_POOL_PER_INITIATOR - each initiatorwill have dedicated threads pool
  • SCST_THREADS_POOL_SHARED - all connected initiators will useshared threads pool
• int (*attach) (struct scst_device *dev) - called when newdevice is being attached to the device handler
• void (*detach) (struct scst_device *dev) - called when newdevice is being detached from the device handler
• int (*attach_tgt) (struct scst_tgt_device *tgt_dev) - calledwhen new tgt_dev (session) is being attached to the device handler
• void (*detach_tgt) (struct scst_tgt_device *tgt_dev) - calledwhen tgt_dev (session) is being detached from the device handler
• int (*parse) (struct scst_cmd *cmd, const struct scst_info_cdb*cdb_info) - called to parse CDB from the cmd and initializecmd->bufflen and cmd->data_direction (both - REQUIRED). Returns thecommand's next state or SCST_CMD_STATE_DEFAULT, if the next defaultstate should be used, or SCST_CMD_STATE_NEED_THREAD_CTX if the functioncalled in atomic context, but requires sleeping, or SCST_CMD_STATE_STOPif the command should not be further processed for now. In theSCST_CMD_STATE_NEED_THREAD_CTX case the function will be recalled in thethread context, where sleeping is allowed. Pay attention to 'atomic'attribute of the cmd, which can be get by scst_cmd_atomic(). It is trueif the function called in the atomic (non-sleeping) context. Must bedefined.
• int (*alloc_data_buf) (struct scst_cmd *cmd) - this functionallows dev handler to handle data buffer allocations on its own. Returnsthe command's next state or SCST_CMD_STATE_DEFAULT, if thenext default state should be used, orSCST_CMD_STATE_NEED_THREAD_CTX if the function called in atomiccontext, but requires sleeping, or SCST_CMD_STATE_STOP if thecommand should not be further processed for now. In theSCST_CMD_STATE_NEED_THREAD_CTX case the function will be recalled in thethread context, where sleeping is allowed. Pay attention to 'atomic'attribute of the cmd, which can be get by scst_cmd_atomic(). It is trueif the function called in the atomic (non-sleeping) context.
• int (*exec) (struct scst_cmd *cmd) - called to execute CDB.Useful, for instance, to implement data caching. The result of CDBexecution is reported via cmd->scst_cmd_done() callback.Returns:
  • SCST_EXEC_COMPLETED - the cmd is done, go to other ones
  • SCST_EXEC_NOT_COMPLETED - the cmd should be sent to SCSImid-level.
  If this function provides sync execution, you should setexec_sync flag and consider to setup dedicated threads bysetting threads_num > 0.Optional, if not set, the commands will be sent directly to SCSIdevice.If this function is implemented, scst_check_local_events() shall becalled inside it just before the actual command's execution.
• int (*dev_done) (struct scst_cmd *cmd) - called to notifydevice handler about the result of the command's execution and performsome post processing. If parse() function is called, dev_done() isguaranteed to be called as well. The command's fieldstgt_resp_flags and resp_data_len should be set by thisfunction, but SCST offers good defaults. Pay attention to 'atomic'attribute of the command, which can be get via scst_cmd_atomic(). It istrue if the function called in the atomic (non-sleeping) context.Returns the command's next state or SCST_CMD_STATE_DEFAULT, ifthe next default state should be used, orSCST_CMD_STATE_NEED_THREAD_CTX if the function called in atomiccontext, but requires sleeping. In the last case, the function will berecalled in the thread context, where sleeping is allowed.
• void (*on_free_cmd) (struct scst_cmd *cmd) - called to notifydevice handler that the command is about to be freed. Could be called onIRQ context.
• int (*task_mgmt_fn) (struct scst_mgmt_cmd *mgmt_cmd, structscst_tgt_dev *tgt_dev) - called to execute a task management command.Returns:
  • SCST_MGMT_STATUS_SUCCESS - the command is donewith success, no further actions required
  • SCST_MGMT_STATUS_* - the command is failed, no further actions required
  • SCST_DEV_TM_NOT_COMPLETED - regular standard actionsfor the command should be done
  NOTE: for SCST_ABORT_TASK it is called under spinlock!
• int (*read_proc) (struct seq_file *seq, struct scst_tgt*tgt), int (*write_proc) (char *buffer, char **start, off_t offset,int length, int *eof, struct scst_tgt *tgt) - those functions can beused to export the driver's statistics and other infos to the worldoutside the kernel as well as to get some management commands from it.If the driver needs to create additional files in its /procsubdirectory, it can use scst_proc_get_dev_type_root() function toget the root proc_dir_entry.

5.2 Device specific drivers registration

scst_register_dev_driver()

To work with SCST a device specific driver must register itself in SCST bycalling scst_register_dev_driver(). It is defined as the following:

Where:

Scsi Raid Array

• dev_type - device specific driver's description structure

The function returns 0 on success or appropriate error code otherwise.

scst_register_virtual_device()

To create a virtual device a device handler must register it in SCST bycalling scst_register_virtual_device(). It is defined as the following:

Where:

• dev_handler - device specific driver's description structure
• dev_name - the new device name, NULL-terminated string. Must be uniqueamong all virtual devices in the system.

The function returns ID assigned to the device on success, or negativevalue otherwise.

All local real SCSI devices will be registered and unregistered by theSCST core automatically, so pass-through dev handlers don't have toworry about it.

5.3 Device specific drivers unregistration

scst_unregister_virtual_device()

Virtual devices unregistered by callingscst_unregister_virtual_device(). It is defined as the following:

Where:

• id - the device's ID, returned by the registration function.

scst_unregister_dev_driver()

Device specific driver is unregistered by callingscst_unregister_dev_driver(). It is defined as the following:

Where:

• dev_type - device specific driver's description structure

6. SCST sessions

6.1 SCST sessions registration

When target driver determines that it needs to create new SCST session(for example, by receiving new TCP connection), it should callscst_register_session(), that is defined as the following:

Where:

• tgt - target
• atomic - true, if the function called in the atomic context
• initiator_name - remote initiator's name, any NULL-terminatedstring, e.g. iSCSI name, which used as the key to found appropriateaccess control group. Could be NULL, then 'default' group is used. Thegroups are set up via /proc interface.
• tgt_priv - pointer to target driver's private data
• result_fn_data - data that will be used as the secondparameter for bfresult_fn/()/ function
• result_fn - pointer to the function that will beasynchronously called when session initialization finishes. Can be NULL.Parameters:
  • sess - session
  • data - target driver supplied to scst_register_session() data
  • result - session initialization result, 0 on success orappropriate error code otherwise

A session creation and initialization is a complex task, which requiressleeping state, so it can't be fully done in interrupt context.Therefore the 'bottom half' of it, if scst_register_session() iscalled from atomic context, will be done in SCST thread context. In thiscase scst_register_session() will return not completely initializedsession, but the target driver can supply commands to this session viascst_rx_cmd(). Those commands processing will be delayed insideSCST until the session initialization is finished, then their processingwill be restarted. The target driver will be notified about finish ofthe session initialization by function result_fn(). On success thetarget driver could do nothing, but if the initialization fails, thetarget driver must ensure that no more new commands being sent or willbe sent to SCST after result_fn() returns. All already sent to SCSTcommands for failed session will be returned in xmit_response()with BUSY status. In case of failure the driver shall callscst_unregister_session() inside result_fn(), it will NOT becalled automatically.

Thus, scst_register_session() can be safely called from IRQ context.

6.2 SCST sessions unregistration

SCST session unregistration basically is the same, except that instead ofatomic parameter there is wait one.

Where:

• sess - session to be unregistered
• wait - if true, instructs to wait until all commands, whichcurrently being executed in the session, finished. Otherwise, targetdriver should be prepared to receive xmit_response() for thesession after scst_unregister_session() returns.
• unreg_done_fn - pointer to the function that will beasynchronously called when the last session's command finishes and thesession is about to be completely freed. Can be NULL. Parameter:
  • sess - session

All outstanding commands will be finished regularly. Afterscst_unregister_session() returned no new commands must be sent to SCSTvia scst_rx_cmd(). Also, the caller must ensure that no scst_rx_cmd() orscst_rx_mgmt_fn_*() is called in parallel withscst_unregister_session().

Function scst_unregister_session()/ can be called before result_fn() ofscst_register_session() called, i.e. during the sessionregistration/initialization.

7. Commands processing and interaction between SCST core and its drivers

Consider simplified commands processing example. It assumes that targetdriver doesn't need own memory allocation, i.e. not definedalloc_data_buf() callback. Example of such target driver is qla2x00t.

The commands processing by SCST started when target driver callsscst_rx_cmd(). This function returns SCST's command. Then thetarget driver finishes the command's initialization, for example,storing necessary target driver specific data there, and callsscst_cmd_init_done() telling SCST that it can start the command processing.Then SCST translates the command's LUN to local device, determines thecommand's data direction and required data buffer size by callingappropriate device handler's parse() callback function. Then:

• If the command required no data transfer, it will be passed toSCSI mid-level directly or via device handler's exec() callback.
• If the command is a READ command (data to the remote/local initiator),necessary space will be allocated and then the command will be passedto SCSI mid-level directly or via device handler's exec() callback.
• If the command is a WRITE command (data from the remote/local initiator),necessary space will be allocated, then the target's rdy_to_xfer()callback will be called, telling the target that the space is ready andit can start data transferring. When all the data are read from thetarget, it will call scst_rx_data(), and the command will be passedto SCSI mid-level directly or via device handler's exec() callback.

When the command is finished by SCSI mid-level, device handler'sdev_done() callback is called to notify it about the command'scompletion. Then in order to send its response the target'sxmit_response() callback is called. When the response, includingdata, if any, is transmitted, the target will callscst_tgt_cmd_done() to tell SCST that it can free the command andits data buffer.

Then during the command's deallocation device handler's and the target'son_free_cmd() callback will be called in this order, if set.

This sequence is illustrated on Figure 2. To simplify the picture, sign'...' means SCST's waiting state for the corresponding command tocomplete. During this state SCST and its drivers continue processing ofother commands, if there are any. One way arrow, for example toxmit_response(), means that after this function returns, nothingvaluable for the current command will be done and SCST goes sleeping orto the next command processing until the corresponding event happens.

7.1 The commands processing functions

scst_rx_cmd()

Function scst_rx_cmd() creates and sends new command to SCST. Returnsthe command on success or NULL otherwise. It is defined as thefollowing:

Where:

• sess - SCST's session
• lun - pointer to device's LUN as specified by SAM in withoutany byte order translation. Extended addressing method is not supported.
• lun_len - LUN's length
• cdb - SCSI CDB
• cdb_len - CDB's length. Can be up to 64KB long.
• atomic - if true, the command will be allocated withGFP_ATOMIC flag, otherwise GFP_KERNEL will be used

scst_cmd_init_done()

Function scst_cmd_init_done() notifies SCST that the driver finishedits part of the command initialization, and the command is ready forexecution. It is defined as the following:

Where:

• cmd - the command
• pref_context - preferred command execution context. SeeSCST_CONTEXT_* constants below for details.

scst_rx_data()

Function scst_rx_data() notifies SCST that the driver received allthe necessary data and the command is ready for further processing. Itis defined as the following:

Where:

• cmd - the command
• status - completion status, see below.
• pref_context - preferred command execution context. SeeSCST_CONTEXT_* constants below for details.

Parameter status can have one of the following values:

• SCST_RX_STATUS_SUCCESS - success
• SCST_RX_STATUS_ERROR - data receiving finished with error, soSCST should set the sense and finish the command by callingxmit_response()
• SCST_RX_STATUS_ERROR_SENSE_SET - data receiving finished with error and the sense is set, so SCST should finish the command by calling xmit_response()
• SCST_RX_STATUS_ERROR_FATAL - data receiving finished withfatal error, so SCST should finish the command, but don't callxmit_response(). In this case the driver must free all associatedwith the command data before calling scst_rx_data().

scst_tgt_cmd_done()

Function scst_tgt_cmd_done() notifies SCST that the driver has sentthe data and/or response. It must not been called if there are an errorand xmit_response() returned something other, than SCST_TGT_RES_SUCCESS.It is defined as the following:

Where:

• cmd - the command
• pref_context - preferred command execution context. SeeSCST_CONTEXT_* constants below for details.

7.2 The commands processing context

Execution context often is a major problem in the kernel driversdevelopment, because many contexts, like IRQ context, greatly limitavailable functionality, therefore require additional complex code inorder to pass processing to more simple context. SCST does its best toundertake most of the context handling.

On the initialization time SCST creates for internal command processingas many threads as there are processors in the system or specified byuser via scst_threads module parameter. Similarly, as many taskletscreated as there are processors in the system.

Each command can be processed in one of four contexts:

1. Directly, i.e. in the caller's context, without limitations
2. Directly atomically, i.e. with sleeping forbidden
3. In the SCST's internal threads
4. In the SCST's per processor tasklets

The target driver sets this context as pref_context parameter for SCSTfunctions. Additionally, target's template's xmit_response_atomicand rdy_to_xfer_atomic flags have direct influence on the context.If one of them is false, the corresponding function will never be calledin the atomic context and, if necessary, the command will be rescheduledto one of the SCST's threads.

SCST in some circumstances can change preferred context to lessrestrictive one, for example, for large data buffer allocation, ifthere is not enough GFP_ATOMIC memory.

Preferred context constants

There are the following preferred context constants:

• SCST_CONTEXT_DIRECT - sets direct command processing (i.e.regular function calls in the current context) sleeping is allowed, nocontext restrictions. Supposed to be used when calling from threadcontext where no locks are held and the driver's architecture allowssleeping without performance degradation or anything like that.
• SCST_CONTEXT_DIRECT_ATOMIC - sets direct command processing (i.e. regular function calls in the current context), sleeping is notallowed. Supposed to be used when calling on thread context where thereare locks held, when calling on softirq context or the driver'sarchitecture does not allow sleeping without performance degradation oranything like that.
• SCST_CONTEXT_TASKLET - tasklet or thread context required for the command processing. Supposed to be used when calling from IRQcontext.
• SCST_CONTEXT_THREAD - thread context required for thecommand processing. Supposed to be used if the driver's architecturedoes not allow using any of above.
• SCST_CONTEXT_SAME - context is the same as it was inprevious call of the corresponding callback. For example, if devhandler's exec() does sync. data reading this value should be used forscst_cmd_done(). The same is true if scst_tgt_cmd_done() called directlyfrom target driver's xmit_response(). Not allowed inscst_cmd_init_done() and scst_cmd_init_stage1_done().

7.3 SCST commands' processing states

There are the following processing states, which a SCST command passesthrough during execution and which could be returned by device handler'sparse() and dev_done() (but not all states are allowed to bereturned):

• SCST_CMD_STATE_INIT_WAIT - the command is created, but scst_cmd_init_done() not called
• SCST_CMD_STATE_INIT - LUN translation (i.e. cmd->tgt_devassignment) state
• SCST_CMD_STATE_PARSE - device handler's parse() is goingto be called
• SCST_CMD_STATE_PREPARE_SPACE - allocation of the command'sdata buffer
• SCST_CMD_STATE_PREPROCESSING_DONE_CALLED - waiting for scst_restart_cmd()
• SCST_CMD_STATE_RDY_TO_XFER - target driver'srdy_to_xfer() is going to be called
• SCST_CMD_STATE_DATA_WAIT - waiting for data from the initiator(until scst_rx_data() called)
• SCST_CMD_STATE_TGT_PRE_EXEC - target driver'spre_exec() is going to be called
• SCST_CMD_STATE_SEND_FOR_EXEC - the command is going to besent for execution
• SCST_CMD_STATE_EXECUTING - waiting for the command's execution finish
• SCST_CMD_STATE_LOCAL_EXEC - the command is being checked ifit should be executed locally
• SCST_CMD_STATE_REAL_EXEC - the command is ready for execution
• SCST_CMD_STATE_REAL_EXECUTING - waiting for CDB's executionfinish
• SCST_CMD_STATE_PRE_DEV_DONE - internal post-exec checks
• SCST_CMD_STATE_MODE_SELECT_CHECKS - internal MODE SELECTpages related checks
• SCST_CMD_STATE_DEV_DONE - device handler's dev_done() isgoing to be called
• SCST_CMD_STATE_PRE_XMIT_RESP - checks before target driver'sxmit_response() is called
• SCST_CMD_STATE_XMIT_RESP - target driver'sxmit_response() is going to be called
• SCST_CMD_STATE_XMIT_WAIT - waiting for data/response'stransmission finish (until scst_tgt_cmd_done() called)
• SCST_CMD_STATE_FINISHED - the command finished and going to be freed

Scsi Disk Device

8. Task management functions

There are the following task management functions supported:

• SCST_ABORT_TASK - this is ABORT_TASK SAM taskmanagement function. Aborts the specified task (command).
• SCST_ABORT_TASK_SET - this is ABORT_TASK_SET SAM taskmanagement function. Aborts all tasks (commands) in the specifiedsession.
• SCST_CLEAR_ACA - this is CLEAR_ACA SAM task managementfunction. Currently does nothing.
• SCST_CLEAR_TASK_SET - this is CLEAR_TASK_SET SAM taskmanagement function. Clears task set of commands on the specifieddevice or session.
• SCST_LUN_RESET - this is LUN_RESET SAM task managementfunction. Resets specified device.
• SCST_TARGET_RESET - this is TARGET_RESET SAM task managementfunction. Resets all devices visible in this session.
• SCST_NEXUS_LOSS_SESS - SCST extension. Notifies about I_Tnexus loss event in the corresponding session. Aborts all tasks there,resets the reservation, if any, and sets up the I_T Nexus loss UA.
• SCST_ABORT_ALL_TASKS_SESS - SCST extension. Aborts alltasks in the corresponding session.
• SCST_NEXUS_LOSS - SCST extension. Notifies about I_T nexusloss event. Aborts all tasks in all sessions of the tgt, resets thereservations, if any, and sets up the I_T Nexus loss UA.
• SCST_ABORT_ALL_TASKS - SCST extension. Aborts all tasks inall sessions of the tgt.

All task management functions return completion status viatask_mgmt_fn_done() when the affected SCSI commands (tasks) areactually aborted, i.e. guaranteed never be executed any time later.

8.1 scst_rx_mgmt_fn_tag()

Function scst_rx_mgmt_fn_tag() tells SCST to perform the specifiedtask management function, based on the command's tag. Can be used onlyfor SCST_ABORT_TASK.

It is defined as the following:

Where:

• sess - the session, on which the command should be performed.
• fn - task management function, one of the constants above.
• tag - the command's tag.
• atomic - true, if the function called in the atomic context.
• tgt_priv - pointer to the target driver specific data, canbe retrieved in task_mgmt_fn_done() via scst_mgmt_cmd_get_status()function.

Returns 0 if the command was successfully created and scheduled forexecution, error code otherwise. On success, the completion status ofthe command will be reported asynchronously via task_mgmt_fn_done()driver's callback.

8.2 scst_rx_mgmt_fn_lun()

Function scst_rx_mgmt_fn_lun() tells SCST to perform the specifiedtask management function, based on the LUN. Currently it can be used forany function, except SCST_ABORT_TASK.

It is defined as the following:

Where:

• sess - the session, on which the command should be performed.
• fn - task management function, one of the constants above.
• lun - LUN, the format is the same as for scst_rx_cmd().
• lun_len - LUN's length.
• atomic - true, if the function called in the atomic context.
• tgt_priv - pointer to the target driver specific data, canbe retrieved in task_mgmt_fn_done() via scst_mgmt_cmd_get_status()function.

Returns 0 if the command was successfully created and scheduled forexecution, error code otherwise. On success, the completion status ofthe command will be reported asynchronously via task_mgmt_fn_done()driver's callback.

Possible status constants which can be returned byscst_mgmt_cmd_get_status():

• SCST_MGMT_STATUS_SUCCESS - success
• SCST_MGMT_STATUS_TASK_NOT_EXIST - requested task does not exist
• SCST_MGMT_STATUS_LUN_NOT_EXIST - requested LUN does not exist
• SCST_MGMT_STATUS_FN_NOT_SUPPORTED - requested TM functiondoes not exist.
• SCST_MGMT_STATUS_REJECTED - TM function rejected.
• SCST_MGMT_STATUS_FAILED - TM function failed.

SCST SGV cache is a memory management subsystem in SCST. One can call ita 'memory pool', but Linux kernel already have a mempool interface,which serves different purposes. SGV cache provides to SCST core, targetdrivers and backend dev handlers facilities to allocate, build and cacheSG vectors for data buffers. The main advantage of it is the cachingfacility, when it doesn't free to the system each vector, which is notused anymore, but keeps it for a while (possibly indefinitely) to let itbe reused by the next consecutive command. This allows to:

• Reduce commands processing latencies and, hence, improve performance;
• Make commands processing latencies predictable, which is essentialfor RT applications.

The freed SG vectors are kept by the SGV cache either for some (possiblyindefinite) time, or, optionally, until the system needs more memory andasks to free some using the set_shrinker() interface. Also the SGV cacheallows to:

• Cluster pages together. 'Cluster' means merging adjacent pages in asingle SG entry. It allows to have less SG entries in the resulting SGvector, hence improve performance handling it as well as allow towork with bigger buffers on hardware with limited SG capabilities.
• Set custom page allocator functions. For instance, scst_user devicehandler uses this facility to eliminate unneeded mapping/unmapping ofuser space pages and avoid unneeded IOCTL calls for buffers allocations.In fileio_tgt application, which uses a regular malloc() function toallocate data buffers, this facility allows 30% less CPU load andconsiderable performance increase.
• Prevent each initiator or all initiators altogether to allocate toomuch memory and DoS the target. Consider 10 initiators, which can haveaccess to 10 devices each. Any of them can queue up to 64 commands, eachcan transfer up to 1MB of data. So, all of them in a peak can allocateup to 10*10*64 = 6.5GB of memory for data buffers. This amount must belimited somehow and the SGV cache performs this function.

9.1 Implementation

From implementation POV the SGV cache is a simple extension of the kmemcache. It can work in 2 modes:

1. With fixed size buffers.
2. With a set of power 2 size buffers. In this mode each SGV cache(struct sgv_pool) has SGV_POOL_ELEMENTS (11 currently) of kmem caches.Each of those kmem caches keeps SGV cache objects (struct sgv_pool_obj)corresponding to SG vectors with size of order X pages. For instance,request to allocate 4 pages will be served from kmem cache[2], since theorder of the of number of requested pages is 2. If later request toallocate 11KB comes, the same SG vector with 4 pages will be reused (seebelow). This mode is in average allows less memory overhead comparingwith the fixed size buffers mode.

Consider how the SGV cache works in the set of buffers mode. When arequest to allocate new SG vector comes, sgv_pool_alloc() viasgv_get_obj() checks if there is already a cached vector with thatorder. If yes, then that vector will be reused and its length, ifnecessary, will be modified to match the requested size. In the aboveexample request for 11KB buffer, 4 pages vector will be reused andmodified using trans_tbl to contain 3 pages and the last entry will bemodified to contain the requested length - 2*PAGE_SIZE. If there is nocached object, then a new sgv_pool_obj will be allocated from thecorresponding kmem cache, chosen by the order of number of requestedpages. Then that vector will be filled by pages and returned.

In the fixed size buffers mode the SGV cache works similarly, exceptthat it always allocate buffer with the predefined fixed size. I.e.even for 4K request the whole buffer with predefined size, say, 1MB,will be used.

In both modes, if size of a request exceeds the maximum allowed forcaching buffer size, the requested buffer will be allocated, but notcached.

Freed cached sgv_pool_obj objects are actually freed to the systemeither by the purge work, which is scheduled once in 60 seconds, or insgv_shrink() called by system, when it's asking for memory.

9.2 Interface

sgv_pool *sgv_pool_create()

This function creates and initializes an SGV cache. It has the followingarguments:

• name - the name of the SGV cache
• clustered - sets type of the pages clustering. The type can be:
  • sgv_no_clustering - no clustering performed.
  • sgv_tail_clustering - a page will only be merged with the latestpreviously allocated page, so the order of pages in the SG will bepreserved
  • sgv_full_clustering - free merging of pages at any place inthe SG is allowed. This mode usually provides the best mergingrate.
• single_alloc_pages - if 0, then the SGV cache will work in the set ofpower 2 size buffers mode. If >0, then the SGV cache will work in thefixed size buffers mode. In this case single_alloc_pages sets thesize of each buffer in pages.
• shared - sets if the SGV cache can be shared between devices or not.The cache sharing allowed only between devices created inside the sameaddress space. If an SGV cache is shared, each subsequent call ofsgv_pool_create() with the same cache name will not create a new cache,but instead return a reference to it.
• purge_interval - sets the cache purging interval. I.e. an SG bufferwill be freed if it's unused for time t purge_interval <= t <2*purge_interval. If purge_interval is 0, then the default intervalwill be used (60 seconds). If purge_interval <0, then the automaticpurging will be disabled. Shrinking by the system's demand will alsobe disabled.

Returns the resulting SGV cache or NULL in case of any error.

void sgv_pool_del()

This function deletes the corresponding SGV cache. If the cache isshared, it will decrease its reference counter. If the reference counterreaches 0, the cache will be destroyed.

void sgv_pool_flush()

This function flushes, i.e. frees, all the cached entries in the SGVcache.

void sgv_pool_set_allocator()

This function allows to set for the SGV cache a custom pages allocator. Forinstance, scst_user uses such function to supply to the cache mapped fromuser space pages.

alloc_pages_fn() has the following parameters:

• sg - SG entry, to which the allocated page should be added.
• gfp - the allocation GFP flags
• priv - pointer to a private data supplied to sgv_pool_alloc()

This function should return the allocated page or NULL, if no page wasallocated.

free_pages_fn() has the following parameters:

• sg - SG vector to free
• sg_count - number of SG entries in the sg
• priv - pointer to a private data supplied to thecorresponding sgv_pool_alloc()

struct scatterlist *sgv_pool_alloc()

This function allocates an SG vector from the SGV cache. It has thefollowing parameters:

• pool - the cache to alloc from
• size - size of the resulting SG vector in bytes
• gfp_mask - the allocation mask
• flags - the allocation flags. The following flags are possible andcan be set using OR operation:
  1. SGV_POOL_ALLOC_NO_CACHED - the SG vector must not be cached.
  2. SGV_POOL_NO_ALLOC_ON_CACHE_MISS - don't do an allocation on acache miss.
  3. SGV_POOL_RETURN_OBJ_ON_ALLOC_FAIL - return an empty SGV object,i.e. without the SG vector, if the allocation can't be completed.For instance, because SGV_POOL_NO_ALLOC_ON_CACHE_MISS flag set.
• count - the resulting count of SG entries in the resulting SG vector.
• sgv - the resulting SGV object. It should be used to free theresulting SG vector.
• mem_lim - memory limits, see below.
• priv - pointer to private for this allocation data. This pointer willbe supplied to alloc_pages_fn() and free_pages_fn() and can beretrieved by sgv_get_priv().

This function returns pointer to the resulting SG vector or NULL in caseof any error.

void sgv_pool_free()

This function frees previously allocated SG vector, referenced by SGVcache object sgv.

void *sgv_get_priv(struct sgv_pool_obj *sgv)

This function allows to get the allocation private data for this SGVcache object sgv. The private data are set by sgv_pool_alloc().

void scst_init_mem_lim()

This function initializes memory limits structure mem_lim according tothe current system configuration. This structure should be latter usedto track and limit allocated by one or more SGV caches memory.

9.3 Runtime information and statistics.

SGV cache runtime information and statistics is available in/proc/scsi_tgt/sgv.

10. Target driver qla2x00t

Target driver qla2x00t allows to use QLogic 2xxx based adapters inthe target (server) mode.

It consists from two parts:

• qla2xxx - patched initiator driver from Linux kernel, whichis, among other things, intended to perform all the initialization andshutdown tasks.
• qla2x00tgt - target mode add-on for the changed qla2xxx

The initiator driver qla2xxx was changed to:

• To provide support for the target mode add-on via a set ofexported callbacks
• To provide extra info and management interface in the driver'ssysfs interface (attributes target_mode_enabled, ports_database, etc.)
• To fix some problems uncovered during target mode development andusage.

The changes are relatively small (few thousands lines big patch) and local.

The changed qla2xxx is still capable to work as initiator only. Mode,when a host acts as initiator and target simultaneously, is supported aswell.

Since firmware interface for 24xx+ chips is fundamentally different fromearlier versions, qla2x00t generally contains 2 separate drivers sharingsome common processing.

10.1 Driver initialization

On initialization, qla2x00tgt registers its SCST template tgt2x_templatein the SCST core. Then during template registration SCST core callsdetect() callback which is function q2t_target_detect().

In this function qla2x00tgt registers its callbacks in qla2xxx bycalling qla2xxx_tgt_register_driver(). Qla2xxx_tgt_register_driver()stores pointer to the being registered callbacks in variable qla_target.

Then q2t_target_detect() calls qla2xxx_add_targets(), which calls foreach known local FC port (HBA instance) qla_target.tgt_host_action()callback with ADD_TARGET action. Then q2t_host_action() callsq2t_add_target() which registers SCST target for this FC port.

If later a new FC port is hot added, qla2x00_probe_one() will also callfor all new local ports qla_target.tgt_host_action() with ADD_TARGETaction.

10.2 Driver unload

When a local FC port is being removed, the Linux kernel callsqla2x00_remove_one(), which then qla_target.tgt_host_action() withREMOVE_TARGET action.

Then q2t_host_action() calls q2t_remove_target(), which unregisters thecorresponding SCST target in SCST. During unregistration SCST core calls release() callback of tgt2x_template, which is q2t_target_release().

Then q2t_target_release() calls q2t_target_stop(). Thenq2t_target_stop() marks this target as stopped by setting flag tgt_stop.When this flag is set, all incoming from initiators commands arerefused.

Then q2t_target_stop() schedules deletion of all sessions of the target.

Then q2t_target_stop() waits until all outstanding commands finished andsessions deleted.

Then q2t_target_stop(), if necessary, calls qla2x00_disable_tgt_mode()to disables target mode, which disables target mode of the correspondingHBA and resets it. Then qla2x00_disable_tgt_mode() waits until resetfinished.

Then q2t_target_stop() returns and then q2t_target_release() frees thetarget.

If module qla2x00tgt is being unloaded, q2t_exit() at first takesq2t_unreg_rwsem on writing. Taking it is necessary to make sure thatq2t_host_action() will not be active during qla2x00tgt unload.

Then q2t_exit() calls scst_unregister_target_template() fortgt2x_template, which then in a loop will unregister all QLA SCST targetsfrom SCST as described above.

10.3 Enabling target mode

When command to enable target mode received,qla_target.tgt_host_action() with action ENABLE_TARGET_MODE called. Thenq2t_host_action() goes over all discovered remote of the being enabledtarget and adds SCST sessions for all them.

Then it calls qla2x00_enable_tgt_mode(), which enables target mode ofthe corresponding HBA and resets it. Then qla2x00_enable_tgt_mode()waits until reset finished.

During reset firmware initialization functions detect that target modeis enables and initialize the firmware accordingly.

10.4 Disabling target mode

When command to disable target mode received,qla_target.tgt_host_action() with action DISABLE_TARGET_MODE called. Thenq2t_host_action() calls q2t_target_stop(), which processes as describe above.

10.5 SCST sessions management

As required by SCSI and FC standards, each remote initiator FC porthas the corresponding SCST session.

Since qla2xxx is not intended to strictly maintain database of remoteinitiator FC ports as it is needed for target mode, qla2x00t uses mixedapproach for SCST sessions management, when both qla2xxx and QLogicfirmware generate events and information about currently active remoteFC ports.

Remote FC ports management also has to handle changing FC and loop IDsafter fabric events, so it needs to constantly monitor FC and loop IDsof the registered FC ports. This is implemented by checks inq2t_create_sess() that being registered FC port already has SCST sessionand q2t_check_fcport_exist() in q2t_del_sess_work_fn(). See below formore info.

Interaction with qla2xxx is implemented using tgt_fc_port_added() andtgt_fc_port_deleted() qla_target's callbacks.

Callback tgt_fc_port_added() called by qla2xxx when the target driverdetects new remote FC port. Assigned to it q2t_fc_port_added() checks ifan SCST session already exists for this remote FC port and, if not,creates it.

Callback tgt_fc_port_deleted() called by qla2xxx when it deletes aremote FC port from its database. Assigned to it q2t_fc_port_deleted()checks if an SCST session already exists for this remote FC port and, ifyes, schedules it for deletion.

Driver qla2x00tgt has 2 types of SCST sessions: local and not local.Sessions created by q2t_fc_port_added() are not local. Local sessionscreated if qla2x00tgt receives a command from remote initiator for whichthere is no know remote FC port and, hence, SCST session. Local sessionsare created in tgt->sess_work (q2t_sess_work_fn()) by callingq2t_make_local_sess(). All received from remote initiators commands forlocal sessions are delayed until the sessions are created.

To minimize affecting initiators by FC fabric events, qla2x00tgt doesn'timmediately delete SCST sessions scheduled for deletion, but insteaddelay them for some time. If during this time a command from an unknownremote initiator received, q2t_make_local_sess()/q2t_create_sess() atfirst check if a session for this initiator already exists and, if yes,undelete then reuse it after updating its s_id and loop_id to new values.

If a session not reused during the delete delay time, thenq2t_del_sess_work_fn() asks the firmware internal database if it knowsthe corresponding remote FC port. If yes, then this session is undeletedand its s_id and loop_id updated to new values. If no, the session isdeleted.

10.6 Handling stuck commands

Driver qla2x00tgt defines in tgt2x_template callbackon_hw_pending_cmd_timeout for handling stuck commands inq2t_on_hw_pending_cmd_timeout() function, with max_hw_pending_timetimeout set Q2T_MAX_HW_PENDING_TIME (60 seconds). If the firmwaredoesn't return reply for one or more IOCBs for the corresponding SCSTcommand, SCST core calls this callback.

In this callback all the stuck commands are forcibly finished.

11. Debugging and troubleshooting

SCST core and its drivers provide excessive debugging and loggingfacilities suitable to catch and analyze problems of virtually any level of complexity.

Depending from amount debugging and logging facilities available, thereare 3 types of builds:

• release - has basic amount of logging, suitable for basictracing. Extra checking is disabled in this mode. This is the defaultmode.
• debug - has full amount of logging and extrachecks enabled.Has slower and much bigger binary code, but suitable for advancedtracing and debugging. Also in this mode more logging is enabled bydefault.
• perf - has all logging and extrachecks disables. Intended toperformance measuremens, including measurements of overhead introducedby the logging and extrachecks facilities.

Switch between build modes is done by calling 'make x2y', where 'x' -current build mode and 'y' - desired build mode. For instance, to switchfrom release to debug mode you should run 'make release2debug'.

11.1 Logging levels management

Logging levels management is done using 'trace_level' file located in thedriver's proc interface subdirectory. Each SCST driver has it, except inthe perf build mode. For instance, for SCST core it's located in/proc/scsi_tgt/. For qla2x00t it's located in /proc/scsi_tgt/qla2x00tgt/.

Reading from it you can find currently enabled logging levels.

You can change them by writing in this file, like:

# echo 'add scsi' >/proc/scsi_tgt/trace_level

The following commands are available:

• add trace_level - adds (enables) the corresponding trace level
• del trace_level - deletes (disables) the corresponding trace level
• set mask - sets all trace levels at ones using a mask, e.g.0x1538
• all - enables all trace levels
• none - disables all trace levels
• default - sets all trace levels in the default value
• dump_prs dev_name - dumps Persistent Reservations states fordevice 'dev_name'

The following trace levels are common for all drivers:

• function - enables printing the corresponding function namesfor each logged messages
• line - enables printing the corresponding numbers of line ofcode for each logged message
• pid - enables printing PIDs of the corresponding processesor threads for each logged message
• scsi - enables logging of processed SCSI commands and theirprocessing results
• mgmt - enables logging of processed Task Management functions
• minor - enables logging of minor events, line unknown SCSIcommands or difference between buffer lengths encoded in CDBs andexpected transfer values
• out_of_mem - enables logging of out of memory events
• entryexit - enables logging of functions entry and exit. Notavailable in the release build.
• mem - enables logging of memory allocation and freeing. Notavailable in the release build.
• debug - enables various debug logging messages. Notavailable in the release build.
• buff - enables logging of various buffers contain. Notavailable in the release build.
• sg - enables logging of SG vectors manipulations. Notavailable in the release build.
• mgmt_dbg - enables debug logging of Task Managementfunctions processing. Not available in the release build.
• special - enables logging of 'special' events. Intended totemporary enable logging of some debug messages without enabling thewhole 'debug' level. Not available in the release build.

The following trace levels are additionally available for SCST core:

• scsi_serializing - enables logging of SCSI commands taskattributes processings (SIMPLE, ORDERED, etc.). Not available in therelease build.
• retry - enables logging of retries of rdy_to_xfer() andxmit_response() target drivers callbacks. Not available in the releasebuild.
• recv_bot, send_bot, recv_top, send_top -enables logging of commands buffers on various processing stages. Notavailable in the release build.

11.2 Preparing a debug kernel

SCST logging can produce huge amount of logging, which default kernelconfiguration can't cope with, so it needs some extra adjustments.

For that you should change in lib/Kconfig.debug or init/Kconfigdepending from your kernel version LOG_BUF_SHIFT from '12 21' to '12 25'.

Then you should in your .config set CONFIG_LOG_BUF_SHIFT to 25.

Also, Linux kernel has a lot of helpful debug facilities, like lockdep,which allows to catch various deadlocks, or memory allocation debugging.It is recommended to enable them during SCST debugging.

The following options are recommended to be enabled (available dependingfrom your kernel version): CONFIG_SLUB_DEBUG, CONFIG_PRINTK_TIME,CONFIG_MAGIC_SYSRQ, CONFIG_DEBUG_FS, CONFIG_DEBUG_KERNEL,CONFIG_DEBUG_SHIRQ, CONFIG_DETECT_SOFTLOCKUP, CONFIG_DETECT_HUNG_TASK,CONFIG_SLUB_DEBUG_ON, CONFIG_SLUB_STATS, CONFIG_DEBUG_PREEMPT,CONFIG_DEBUG_RT_MUTEXES, CONFIG_DEBUG_PI_LIST, CONFIG_DEBUG_SPINLOCK,CONFIG_DEBUG_MUTEXES, CONFIG_DEBUG_LOCK_ALLOC, CONFIG_PROVE_LOCKING,CONFIG_LOCKDEP, CONFIG_LOCK_STAT, CONFIG_DEBUG_SPINLOCK_SLEEP,CONFIG_STACKTRACE, CONFIG_DEBUG_BUGVERBOSE, CONFIG_DEBUG_VM,CONFIG_DEBUG_VIRTUAL, CONFIG_DEBUG_WRITECOUNT, CONFIG_DEBUG_MEMORY_INIT,CONFIG_DEBUG_LIST, CONFIG_DEBUG_SG, CONFIG_DEBUG_NOTIFIERS,CONFIG_FRAME_POINTER, CONFIG_FAULT_INJECTION, CONFIG_FAILSLAB,CONFIG_FAIL_PAGE_ALLOC, CONFIG_FAIL_MAKE_REQUEST,CONFIG_FAIL_IO_TIMEOUT, CONFIG_FAULT_INJECTION_DEBUG_FS,CONFIG_FAULT_INJECTION_STACKTRACE_FILTER.

11.3 Preparing logging subsystem

Adaptec Scsi Drivers

It is recommended that you system logger daemon on the target configured:

• To store kernel logs in separate files on the fastest disk youhave. It will be better if this disk is dedicated for logging or, atleast, doesn't contain your LUNs data.
• To write the kernel logs to the disk in asynchronous manner, i.e.without calling fsync() after each written message. Usually, you canachieve it, if you add a '-' sign before the corresponding file path inyour syslog daemon conf file, like:kern.* -/var/log/kern.log

11.4 Decoding OOPS messages

You can decode an OOPS message to the corresponding line in C fileusing gdb 'l' command. For example, an OOPS message has a line:

You can decode it by:

For that the corresponding module (iscsi-scst.ko) should be build withdebug info. But modules not always have debug info built-in. Toworkaround it you can add '-g' flag in the corresponding Makefile(without changing anything else!) or enable in .config using 'makemenuconfig' building kernel with debug info. Then rebuild only the .ofile you need.

Microsoft Scsi Driver

For instance, to decode OOPS in mm/filemap.c in the kernel you needenable in .config building kernel with debug info and then run: